In this paper, we introduce “scientific and empirical adversarial AI attacks” (SEA AI attacks) as umbrella term for not yet prevalent but technically feasible deliberate malicious acts of specifically crafting AI-generated samples to achieve an epistemic distortion in (applied) science or engineering contexts. In view of possible socio-psychotechnological impacts, it seems responsible to ponder countermeasures from the onset on and not in hindsight. In this vein, we consider two illustrative use cases: the example of AI-produced data to mislead security engineering practices and the conceivable prospect of AI-generated contents to manipulate scientific writing processes. Firstly, we contextualize the epistemic challenges that such future SEA AI attacks could pose to society in the light of broader i.a. AI safety, AI ethics and cybersecurityrelevant efforts. Secondly, we set forth a corresponding supportive generic epistemic defense approach. Thirdly, we effect a threat modelling for the two use cases and propose tailor-made defenses based on the foregoing generic deliberations. Strikingly, our transdisciplinary analysis suggests that employing distinct explanation-anchored, trustdisentangled and adversarial strategies is one possible principled complementary epistemic defense against SEA AI attacks - albeit with caveats yielding incentives for future work.
Progress in the AI field unfolds a wide growing array of beneficial societal effects with AI permeating more and more crucial application domains. To forestall ethically-relevant ramifications, research from a variety of disciplines tackling pertinent AI safety [Amodei et al., 2016; Bostrom, 2017; Burden and Herna´ndez-Orallo, 2020; Fickinger et al., 2020; Leike et al., 2017], AI ethics and AI governance issues [Floridi et al., 2018; Jobin et al., 2019; O´ hE´ igeartaigh et al., 2020; Raji et al., 2020] gained momentum at an international level. In addition, cybersecurity-oriented frameworks in AI safety [Aliman et al., 2021; Brundage et al., 2018; Pistono and Yampolskiy, 2016] stressed the necessity to not only address unintentional errors, unforeseen repercussions and bugs in the context of ethical AI design but also AI risks linked to intentional malice i.e. deliberate unethical design, attacks and sabotage by malicious actors. In parallel, the convergence of AI with other technologies increases and diversifies the attack surface available to malevolent actors. For instance, while AI-enhanced cybersecurity opens up novel valuable possibilities for defenders [Zeadally et al., 2020], AI simultaneously provides new affordances for attackers [Ashkenazy and Zini, 2019] from AI-aided social engineering [Seymour and Tully, 2016] to AI-concealed malware [Kirat et al., 2018]. Next to the capacity of AI to extend classical cyberattacks in scope, speed and scale [Kaloudi and Li, 2020], a notable emerging threat is what we denote AI-aided epistemic distortion. The latter represents a form of AI weaponization and is increasingly studied in its currently most salient form, namely AI-aided disinformation [Aliman et al., 2021; Chesney and Citron, 2019; Kaloudi and Li, 2020; Tully and Foster, 2020] which is especially relevant to information warfare [Hartmann and Giles, 2020]. Recently, the weaponization of Generative AI for information operations has been described as “a sincere threat to democracies” [Hartmann and Steup, 2020]. In this paper, we analyze attacks and defenses pertaining to another not yet prevalent but technically feasible and similarly concerning form of AI-aided epistemic distortion with potentially profound societal implications: scientific and empirical adversarial AI attacks (SEA AI attacks).
With SEA AI attacks, we refer to any deliberately
malicious AI-aided epistemic distortion which predominantly and
directly targets (applied) science and technology assets (as
opposed to information operations where a wider societal
target is often selected on ideological/political grounds). In
short, the expression acts as an umbrella term for malicious
actors utilizing or attacking AI at pre- or post-deployment
stages with the deliberate adversarial aim to deceive,
sabotage, slow down or disrupt (applied) science, engineering or
related endeavors. Obviously, SEA AI attacks could be
performed in a variety of modalities
As reflected in the law of requisite variety (LRV) known from cybernetics, “only variety can destroy variety” [Ashby, 1961]. Applied to SEA AI attacks, it signifies that since malicious adversaries are not only exploiting vulnerabilities from a heterogeneous socio-psycho-technological landscape but also specially vulnerabilities of epistemic nature, suitable defense methods may profit from an epistemic stance. Applying the cybernetic LRV offers a valuable domain-general transdisciplinary tool able to stimulate and invigorate novel tailored defenses in a diversity of harm-related problems from cybersecurity [Vinnakota, 2013] to AI safety [Aliman, 2020a] over AI ethics [Ashby, 2020]. In short, utilizing insights from epistemology as complementary basis to frame defense methods against SEA AI attacks seems indispensable. Past work predominantly analyzed countermeasures of socio-psycho-technological nature to combat the spread of (audio-)visual, audio and textual deepfakes as well as “fake news” more broadly. For instance, the technical detection of AI-generated content [Wahle et al., 2021] has been often thematized and even lately applied to “fake news” in the healthcare domain [Baris and Boukhers, 2021]. Furthermore, in the context of counteracting risks posed by the deployment of sophisticated online bots, it has been suggested that “technical solutions, while important, should be complemented with efforts involving informed policy and international norms to accompany these technological developments” and that “it is essential to foster increased civic literacy of the nature of ones interactions” [Boneh et al., 2019]. Another analysis presented a set of defense measures against the spread of deepfakes [Chesney and Citron, 2019] which contained i.a. legal solutions, administrative agency solutions, coercive and covert responses as well as sanctions (when effectuated by state actors) and speech policies for online platforms. Concerning “fake science news” and their impacts on “credibility and reputation of the science community” [Ho et al., 2020], it has been even postulated by Makri that “science is losing its relevance as a source of truth” and “the new focus on post-truth shows there is now a tangible danger that must be addressed” [Makri, 2017]. Following the author, scientists could equip citizens with sense-making tools without which “emotions and beliefs that pander to false certainties become more credible” [Makri, 2017].
While some of those socio-psycho-technological
countermeasures and underlying assumptions are debatable, we
complementarily zoom in different epistemic defenses against
SEA AI attacks being directed against scientific and empirical
frameworks. Amidst an information ecosystem with
quasiomnipresent terms such as “post-truth” or “fake news” and
in light of data-driven research trends embedded within
trustbased infrastructures, it seems daunting to face a threat
landscape populated by AI-generated artefacts such as: 1) “fake
data” and “fake experiments”, 2) “fake research papers”
Firstly, it has been lately propounded that the societal
perception of a “post-truth” era is often linked to the implicit
assumption that truth can be equated with consensus which
is why it seems recommendable to consider a deflationary
account of truth [Bufacchi, 2021] – i.e. where the concept
is for instance strictly reserved to scientifically-relevant
epistemic contexts. On such a deflationary account of truth
disentangled from consensus, it has been argued that even if
consensus and trust seem eroded, we neither inhabit a post-truth
nor a science-threatening post-falsification age [Aliman and
Kester, 2020]. Secondly, we never had a direct access to
physical reality which we could have suddenly lost with the advent
of “fake news”. In fact, as stated by Karl Popper: “Once we
realize that human knowledge is fallible, we realize also that
we can never be completely certain that we have not made a
mistake” [Popper, 1996]. Thirdly, the epistemic aim in
science can neither be truth directly [Frederick, 2020] nor can it
be truer beliefs via justifications. The former is not directly
experienced and the latter has been shown to be logically
invalid by Popper [Popper, 2014]. Science is
quintessentially explanatory i.e. it is based on explanations [Deutsch,
2011] and not merely on data. While the epistemic aim
cannot be certainty or justification
Against the background of this epistemic bedrock, one can now re-assess the threat landscape of SEA AI attacks. Firstly, one can conclude that AI-generated “fake data” and “fake experiments” could slow down but not terminally disrupt scientific and empirical procedures. In the case of misguiding confirmatory data, it has no epistemic effect since as opposed to empiricist epistemology, explanation-anchored science does not utilize any scheme of credence updates for a theory and it is clear that “a severely tested but unfalsified theory may be false” [Frederick, 2020]. In the case of misleading data that is accepted to falsify a theory T , one runs the risk to con1That our epistemic aim can be “truer explanations” or explanations that lead us “closer to the truth” has been sometimes confusingly written by Deutsch and Popper respectively but this type of account requires a semantic refinement [Frederick, 2020]. sider mistakenly that T has been made problematic. However, since it is not permissible to drop T in the absence of a rival theory T 0 representing a better explanation than T , the adverarial capabilities of the SEA AI attacker are limited. In short, theories cannot be deleted from the collective knowledge via such SEA AI attacks without more ado. Secondly, when contemplating the case of AI-generated “fake research papers”, it seems that they could slow down but not disrupt scientific methodology. Overall, one could state that the danger lies in the uptake of deceptive theories. However, theories are only integrated in explanatory-anchored science if they represent better explanations in comparison to alternatives or in the absence of alternatives if they explain novel phenomena. In a nutshell, it takes explanations that are simultaneously misguiding and better for such a SEA AI attack to succeed. This is a high bar for imitative language models if meant to be repeatedly and systematically performed2 and not merely as a unique event by chance. Further, even in the case a deceptive theory has been integrated in a field, that is always only provisionally such that it could be revoked at any suitable moment e.g. once a better explanation arises and repeated experiments falsify its claims. If in the course of this, an actually better explanation had been mistakenly considered as refuted, it can always be re-integrated once this is noticed. In fact, “a falsified theory may be true” [Frederick, 2020] if the accepted observations believed to have falsified it were wrong. Thirdly, when now considering the final case of AI-generated “fake reviews”, it becomes clear that they could similarly slow down but not terminally disrupt the scientific method. At worst some existing theories could be unnecessarily problematized and misguiding theories uptaken, but all these epistemic procedures can be repealed retrospectively.
In short, explanation-anchored science is resilient (albeit not immune) against SEA AI attacks but one can humbly face the idea that it is not because scientists can “tease out falsehood from truths” [Ho et al., 2020], but because explanationanchored science attempts to tease out better from worse explanations while permanently requiring the creation of new ones whereby the steps made can always be revoked, revised and even actively adversarially counteracted. That entails a sort of epistemic dizziness and one can never trust one’s own observations. Also, human mental constructions are inseparably cognitive-affective and science is not detached from social reality [Barrett, 2017]. In our view, for a systematic management of this epistemic dizziness, one may profit from an adversarial approach that permanently brings to mind that one might be wrong. Last but not least, an important feature discussed is that the epistemic aim not being truth (which itself is also not consensus and does not rely on trust to exist) but instead better explanations, none of the mentioned 2That there could exist a task which imitative language models are “theoretically incapable of handling” has been often put into question [Sahlgren and Carlsson, 2021]. However, on epistemic grounds elaborated in-depth previously [Aliman, 2020a; Aliman et al., 2021] which might be amenable to experimental falsifiability [Aliman, 2020b], we assume that the task to consciously create and understand novel yet unknown explanatory knowledge [Deutsch, 2011] – which humans are capable of performing if willing to – cannot be learned by AI systems by mere imitation. methods are dependent on trust per se – making it a trustdisentangled view. To sum up, we identified 3 key generic features for epistemic defenses against SEA AI attacks:
3
In the following Subsection 3.1, we briefly perform an exemplary threat modelling for the two specific use cases introduced in Section 1. The threat model narratives are naturally non-exhaustive and are selected for illustrative purposes to display plausible downward counterfactuals projecting capabilities to the recent counterfactual past in the spirit of cocreation design fictions in AI safety [Aliman et al., 2021]. In Subsection 3.2, we then derive corresponding tailor-made defenses from the generic characteristics that have been carved out in the last Section 2 while thematizing notable caveats. 3.1
Use Case Security Engineering • Adversarial goals: As briefly mentioned in Section 1, CTI (which is information related to cybersecurity threats and threat actors to support analysts and security systems in the detection and mitigation of cyberattacks) can be polluted via misleading AI-generated samples to fool cyber defense systems at the training stage [Ranade et al., 2021]. Among others, CTI is available as unstructured texts but also as knowledge graphs taking CTI texts as input. A textual data poisoning via AI-produced “fake CTI” represents a form of SEA AI attack that was able to succesfully deceive (AI-enhanced) automated cyber defense and even cybersecurity experts which “labeled the majority of the fake CTI samples as true despite their expertise” [Ranade et al., 2021]. It is easily conceivable that malicious actors could specifically tailor such SEA AI attacks in order to subvert cyber defense in the service of subsequent covert time-efficient, micro-targeted and large-scale cybercrime. For 2021, cybercrime damages are estimated to reach 6 trillion USD [Benz and Chatterjee, 2020; Ozkan et al., 2021] making cybercrime a top international risk with a growing set of affordances which malicious actors do not hesitate to enact. Actors interested in “fake CTI” attacks could be financially motivated cybercriminals or state-related actors. Adversarial goals could e.g. be to acquire private data, CTI poisoning in a cybercrime-as-a-service form, gain strategical advantages in cyber operations, conduct espionage or even attack critical infrastructure endangering human lives. • Adversarial knowledge: Since it is the attacker that finetunes the language model generating the “fake CTI” samples for the SEA AI attack, we consider a white box setting for this system. The attacker does not require knowledge about the internal details of the targeted automated cyber defense allowing a black-box setting with regard to this system at training time. In case the attacker directly targets human security analysts by exposing them to misleading CTI, the SEA AI attack can be interpreted as a type of adversarial example on human cognition in a black-box setting. However, in such cases “open-source intelligence gathering and social engineering are exemplary tools that the adversary can employ to widen its knowledge of beliefs, preferences and personal traits exhibited by the victim” [Aliman et al., 2021]. Hence, depending on the required sophistication, a type of grey-box setting is achievable. • Adversarial capabilities: The use of SEA AI attacks could have been useful at multiple stages. CTI text could have been altered in a micro-targeted way offering diverse capacities to a malicious actor: to distract analysts from patching existing vulnerabilities, to gain time for the exploitation of zero-days, to let systems misclassify malign files as benign [Mahlangu et al., 2019] or to covertly take over victim networks. In the light of complex interdependencies, the malicious actor might not even have had a full overview of all repercussions that AI-generated “fake CTI” attacks can engender. Poisoned knowledge graphs could have led to unforeseen domino-effects inducing unknown second-order harm. As long-term strategy, the malicious actor could have harnessed SEA AI attacks on applied science writing to automate the generation of cybersecurity reports (for it to later serve as CTI inputs) corroborating the robustness of actually unsafe defenses to covertly subvert those or simply to spread confusion.
•
Adversarial goals: The emerging issue of (AI-aided) information operations in social media contexts which involves entities related to state actors has gained momentum in the last years [Prier, 2017; Hartmann and Giles, 2020]. A key objective of information operations that has been repeatedly mentioned is the intention to blur what is often termed as the line between facts and fictions [Jakubowski, 2019]. Naturally, when logically applying the epistemic stance introduced in the last Section 2, it seems recommendable to avoid such formulations for clarity since potentially confusing. Hence, we refer to it simply as epistemic distortion. SEA AI attacks on scientific writing being a form of AI-aided epistemic distortion, it could represent a lucrative opportunity for state actors or politically motivated cybercriminals willing to ratchet up information operations. On a smaller scale, other potential malicious goals could also involve companies with a certain agenda for a product that could be threatened by scientific research. Another option could be advertisers that monetize attention via AI-generated research papers in click-bait schemes. • Adversarial knowledge: As in the first use case, the language model is available in a white-box setting. Moreover, since this SEA AI attack directly targets human entities, one can again assume a black-box or grey-box scenario depending on the required sophistication of the attack. For instance, since many scientists utilize social media platforms, open source intelligence gathering on related sources can be utilized to tailor contents. • Adversarial capabilities: In the domain of adversarial machine learning, it has been stressed that for security reasons it is important to also consider adaptive attacks [Carlini et al., 2019], namely reactive attacks that adapt to what the defense did. A malicious actor aware of the discussed explanation-anchored, trustdisentangled and adversarial epistemic defense approach could have exploited a wide SEA AI attack surface in case of no consensus on the utility of this defense. For instance, a polarization between two dichotomously opposed camps in that regard could have offered an ideal breeding ground for divisive information warfare endeavors. For some, the perception of increasing disagreement tendencies may have confirmed post-truth narratives. Not for malicious reasons, but because it was genuinely considered. This in turn could have cemented echo chamber effects now fuelled by a divided set of scientists one part of which considered science to be epistemically defeated. This combined with posttruth narratives and the societal-level automated disconcertion [Aliman et al., 2021] via the mere existence of AI-generated fakery could have destabilized a fragile society and incited violence. Massive and rapid largescale SEA AI attacks in the form of a novel type of scientific astroturfing could have been employed to automatically reinforce the widespread impression of permanently conflicting research results on-demand and tailored to a scientific topic. The concealed or ambiguous AI-generated samples (be it data, experiments, papers or reviews) would not even need to be overrepresented in respected venues but only made salient via social media platforms being one of the main information sources for researchers – a task which could have been automated via social bots influencing trending and sharing patterns. A hinted variant of such SEA AI attacks could have been a flood of confirmatory AI-generated texts that corroborate the robustness of defenses across a large array of security areas in order to exploit any reduced vulnerability awareness. Finally, hyperlinks with attention-driving fake research contribution titles competing with science journalism and redirecting to advertisement pages could have polluted results displayed by search engines. 3.2
As is also the case with other advanced not yet prevalent but technically already feasible AI-aided information operations [Hartmann and Giles, 2020] and cyberattacks targeting AIs [Hartmann and Steup, 2020], consequences could have ranged from severe financial losses to threats to human lives. Multiple socio-psycho-technological solutions including the ones reviewed in Section 1 which may be (partially) relevant to SEA AI attack scenarios have been previously presented. Here, we complementarily focus on the epistemic dimensions one can add to the pool of potential solutions by applying the 3 generic features extracted in Section 2 to both use cases. We also emphasize novel caveats. Concerning the first use case of “fake CTI” SEA AI attacks, the straightforward thought to restrict the use of data from open platforms is not conducive to practicability not only due to the amount of crucial information that a defense might miss, but also because it does not protect from insider threats [Ranade et al., 2021]. However, common solutions such as the AI-based detection of AI-generated outputs or trust-reliant scoring systems to flag trusted sources do not seem sufficient either without more ado since the former may fail in the near future if the generator tends to win and the latter is at risk due to impersonation possibilities that AI itself augments and due to the mentioned insider threats. Interestingly, the issue of malicious insider threats is also reflected in the second use case with scientific writing being open to arbitrary participants.
Defense for Security Engineering Use Case and Caveats
1. Explanation-anchored instead of data-driven: An
explanation-anchored solution can be formulated from
the inside out. Although AI does not understand
explanations, it is thinkable that a technically feasible future
hybrid active intelligent system3 for automated cyber
defense could use knowledge graph inconsistencies
[Heyvaert et al., 2019] as signals to calculate when it will
epistemically seek clarification from a human analyst,
when to actively query differing sources and sensors or
when to follow habitual courses of action. But the
creativity of human malicious actors cannot be predicted
and thus neither the system nor human analysts are able
to prophesy over a space of not yet created attacks. Also,
as long as the system’s sensors are learning-based AI, it
stays an Achilles heel due to the vulnerability to attacks.
2. Trust-disentangled instead of trust-dependent: Such a
procedure could seem disadvantageous given the fast
reactions required in cyber defense. However, an
adversarial explanation-anchored framework is orthogonal to
the trust policy used. Trust-disentangled does not
necessarily signify zero-trust4 at all levels if impracticable.
3. Adversarial instead of (self-)compliant: A permanently
rotating in-house adversarial team is required.
Activities can include red teaming, penetration testing and the
development of (adaptive) attacks i.a. with AI-generated
“fake CTI” text samples. A staggered approach is
cogitable in which automated defense processes that
happen at fast scales (e.g. requiring rapid access to open
source CTI) rely on interim (distributed) trust while all
others – especially those involving human deliberation
to create novel defenses and attacks – strive for
zerotrust information sharing (e.g. via a closed blockchain
with a restricted set of authorized participants having
read and write rights). In this way, one can create an
interconnected 3-layered epistemically motivated
security framework: a slow creative human-run adversarial
3Such a system could instantiate technical self-awareness
[Aliman, 2020a]
4The zero-trust [Kindervag, 2010] paradigm advanced in cybersecurity in the last decade which assumes “that adversaries are already inside the system, and therefore imposes strict access and authentication requirements” [Collier and Sarkis, 2021] seems highly appropriate in this increasingly complex security landscape. counterfactual layer on top of a slow creative humanrun defensive layer steering a very fast hybrid-active-AIaided automated cyber defense layer. Important caveats are that such a framework: 1) can be resilient but not immune, 2) can not and should not be entirely automated. Defense for Science Writing Use Case and Caveats 1. Explanation-anchored instead of data-driven: A practical challenge for SEA AI attacks may seem the need for scientists to agree on pragmatic criteria for “better” explanations (but widely accepted cases are e.g. the preference for “simpler”, “more innovative” and “more interesting” ones). Also, due to automated disconcertion, reviewers could always suspect that a paper was AI-generated (potentially at the detriment of human linguistic statistical outliers). However, this is not a sufficient argument since explanation-anchored science and criticism focus on content and not on source or style.
trust-disentanglement, a paper generated by a presentday AI would not only be rejected on provenance grounds but due to its merely imitative and nonexplanatory content. Though, an important asset is the review process which if infiltrated by imitative AI-generated content could slow down explanationanchored criticism if not thwarted fastly. A zero-trust scheme could mitigate this risk time-efficiently (e.g. via a consortium blockchain for review activities). Another zero-trust method would be to taxonomically monitor SEA AI attack events at an international level e.g. via an AI incident base [McGregor, 2020] tailored to these attacks and complemented by adversarial retrospective counterfactual risk analyses [Aliman et al., 2021] and defensive solutions. The monitoring can be AI-aided (or in the future hybrid-active-AI-aided) but human analysts are indispensable for a deep semantic understanding [Aliman et al., 2021]. In short, also here, we suggest an interconnected 3-layered epistemic framework with adversarial, defensive and hybrid-active-AI-aided elements. 3. Adversarial instead of (self-)compliant: As advanced adversarial strategy which would also require responsible coordinated vulnerability disclosures [Kranenbarg et al., 2018], one could perform red teaming, penetration tests and (adaptive) attacks employing AI-generated “fake data and experiments”, “fake papers” and “fake reviews” [Tallo´n-Ballesteros, 2020]. Candidates for a blue team are e.g. reviewers and editors. Concurrently, urgent AI-related plagiarism issues arise [Dehouche, 2021]. 4
For requisite variety, we introduced a complementary generic epistemic defense against not yet prevalent but technically feasible SEA AI attacks. This generic approach foregrounded explanation-anchored, trust-disentangled and adversarial features that we instantiated within two illustrative use cases involving language models: AI-generated samples to fool security engineering practices and AI-crafted contents to distort scientific writing. For both use cases, we compactly worked out a transdisciplinary and pragmatic 3-layered epistemically motivated security framework composed of adversarial, defensive and hybrid-active-AI-aided elements with two major caveats: 1) it can be resilient but not immune, 2) it can not and should not be entirely automated. In both cases, a proactive exposure to synthetic AI-generated material could foster critical thinking. Vitally, the existence of truth stays a legitimate raison d’eˆtre for science. It is only that in effect, one is not equipped with a direct acces to truth, all observations are theory-laden and what one think one knows is linked to what is co-created in one’s collective enactment of a world with other entities shaping and shaped by physical reality. Thereby, one can craft explanations to try to improve one’s active grip on a field of affordances but it stays an eternal mental tightrope walking of creativity. In view of this inescapable epistemic dizziness, the main task of explanationanchored science is then neither to draw a line between truth and falsity nor between the trusted and the untrusted. Instead, it is to seek to robustly but provisionally separate better from worse explanations. While this steadily renewed societally relevant act does not yield immunity against AI-aided epistemic distortion, it enables resiliency against at-present thinkable SEA AI attacks. To sum up, the epistemic dizziness of conjecturing that one could always be wrong could stimulate intellectual humility, but also unbound(ed) (adversarial) explanatory knowledge co-creation. Future work could study how language AI – which could be exploited for future SEA AI attacks e.g. instrumental in performing cyber(crime) and information operations – could conversely serve as transformative tool to augment anthropic creativity and tackle the SEA AI threat itself. For instance, language AI could be used to stimulate human creativity in future AI and security design fictions for new threat models and defenses. In retrospective, AI is already acting as a catalyst since the very defenses humanity now crafts can broaden, deepen and refine the scope of explanations i.a. also about better explanations – an unceasing but also potentially strengthening safety relevant quest.