Large Language Models (LLMs), and Generative AI (GenAI) more generally, have been the center of much attention in both media and research following recent technical advances. In the wake of the recent surge of users services like ChatGPT and GitHub Copilot have seen, there has also been much discussion on the potential negative impacts of these tools, from job loss to cheating in schools by using AI, to security issues in AI-generated code. To this end, a large number of papers has recently been published on the ethical issues and negative impacts of LLMs, across a wide variety of scientific disciplines. To help tie this recent discussion on the ethical issues of LLMs to the existing discussion on AI ethics, we conduct a Systematic Literature Review (SLR) and review 116 papers. We extract 434 individual ethical issues from these papers, based on which we identify 39 diferent categories of ethical issues. We then map these 39 categories of ethical issues into the seven requirements for trustworthy AI found in the Ethics Guidelines for Trustworthy AI (AI HLEG) in order to understand how these concerns related to the existing discussion on AI ethics. While various new practical issues are identified in the process, on a conceptual level the issues found in LLMs are related to the ones already identified in AI ethics and can be related to existing AI ethics principles. Overall, this SLR provides a summary of the current discussion on ethical issues in LLMs.
CEUR ceur-ws.org Principles
Recent advances in the field of Generative AI (GenAI) have brought GenAI systems into the spotlight in AI ethics as well. For the most part, the focus has been on Large Language Models (LLMs) in particular. These new types of systems have resulted in another surge of discussion on ethical issues and risks associated with ML, especially in the media. We have also noticed a similar surge in research papers focusing on LLMs, with or without a focus on ethical aspects.
LLMs have had a massive impact on making ML systems available for consumer use, primarily
through B2C cloud services that require no technical ML knowhow. For example, ChatGPT
became the fastest growing service in history thus far, surpassing TikTok, Facebook, and
other online services with a history of quick growth [
Much of the existing discussion in the field of AI ethics has focused on the development of
ethical ML systems. E.g., principles that ML systems need to adhere to in order to be ethical (see
for example [
These developments may present some points of reflection for AI ethics researchers. For
example, to what extent are these issues new on a conceptual level? Are there any new issues
related to fairness that should be considered, or is the novelty in the changed practical context?
For example, if we are concerned about the biased output of LLMs, we are ultimately still
concerned about bias just as we were concerned about it in relation to decision-making support
systems. Yet, do these new system use contexts nonetheless present new challenges for AI
ethics research? Given that AI ethics overall is a field ripe with various competing concepts
and definitions [
To better understand the implications of LLMs for the existing AI ethics discussion, we
consider systematically reviewing this recent surge of publications to be beneficial for the field.
Thus, in this paper, we conduct a Systematic Literature Review (SLR) of publications discussing
ethical challenges associated with GenAI, and specifically Large Language Models (LLMs). We
utilize the Ethics Guidelines for Trustworthy AI [
We consider literature reviews related to AI ethics related work in the context of this paper.
Thus, the most closely related paper we have identified is that of Weidinger et al. [
While various other literature (grey or otherwise) reviews related to AI exist, these are
less related to our work in this SLR. Some reviews related to AI ethics include include the
following. Jobin et al. [
This section discusses the SLR protocol. Section 3.1 discusses the search strategy. Section 3.2 discusses the inclusion and exclusion process. Section 3.3 discusses the data extraction.
For this SLR, we searched for literature from three databases: ACM Digital Library, IEEE
Xplore, and Scopus. We employed the following search string for all three, with potential minor
adjustments to adhere to the constraints of the search engine of each database:
”large language model*” OR ”chatgpt” AND ”ethic*” OR ”fair*” OR ”transpar*” OR ”explaina*”
OR ”trustworth*” OR ”human agenc*” OR ”oversight” OR ”privacy” OR ”diversity*” OR ”discrimin*”
AND ”generat*”
The first part of the search string was used to limit the search to these specific types of ML
systems. ChatGPT as a specific search term was included due to being the most popular service
at this time, to the point where it regularly appeared in titles and abstracts of various papers.
The second part of the search string comprises the requirements for ethical AI found in the
Ethics Guidelines for Trustworthy AI (henceforth AI HLEG) [
Inclusion criteria Exclusion criteria In English Not in English Accessible to us Behind a paywall and not
self-archived Peer-reviewed Not peer-reviewed (or not
confirmable) Is a conference paper or a Is a book, book chapter, edijournal article torial, pre-print, letter, note,
or keynote Is focused on GenAI/ LLMs Is not focused on GenAI/
LLMs Is focused on discussing eth- Is not focused on ethical ical issues or contains sub- issues in GenAI/ LLMs, or stantial discussion on them does not contain substantial otherwise discussion on them
The searches were conducted in June 2023. The initial searches resulted in a total of 1647 studies. After this, the results were iteratively evaluated and included or excluded in stages. This proceeded as follows: (1) inclusion/exclusion based on document type and removal of duplicate results (1337 left, 310 excluded) , (2) inclusion/exclusion based on title and abstract (157 left, 1180 excluded) , and (3) inclusion/exclusion based on full text (116 included for review, 41 excluded). The inclusion and exclusion criteria are detailed in Table 1.
For clarity, we opted to only include papers explicitly discussing LLMs, even if general NLP literature could be considered relevant. As for ”substantial discussion” on ethical issues, we decided to consider any paper that discusses ethical issues in the abstract to fulfill this criterion. Studies discussing relevant topics, such as cybersecurity, were excluded if they only focused on technical factors (e.g., attack vectors) without any ethical aspects being discussed in the abstract (or the full paper, in the third round).
The following information was extracted from each of the included 116 final papers: (a) paper type (conference/journal/book and conceptual/empirical/lit. review), (b) types of ethical issues identified, (c) any mitigation measures proposed in relation to the ethical issues discussed, (d) database, (e) title, (f) authors, (g) DOI, (h) publication venue, (i) publication year.
Where possible, we utilized direct citations from the paper to extract the ethical issues and the mitigation proposals. In cases where the argumentation was more spread out, the key points were manually summarized and included in the spreadsheet brackets ([]). After this data had been extracted, the ethical issues and mitigation proposals were further analyzed by allocating them into categories. These categories were iteratively synthesized from the data so that (a) recurring issues could be best highlighted but so that (b) there would be as little overlap as possible between the categories.
Based on the SLR process, we identified 39 categories of ethical issues currently associated with LLMs in existing literature. These categories were synthesized based on 434 individual instances of ethical issues present in the 116 final papers. Figure 1 presents how these 434 issues are distributed between the seven requirements of the AI HLEG. As we report our results, we cite the 116 final papers using the following citation scheme: [Sn] where n is the number of the [S]tudy the way these 116 papers are presented in Appendix 1.
The requirement of human agency and oversight posits that AI should support human autonomy
and decision-making, while also allowing for human oversight of the AI. Human actors should
be able to make informed decisions regarding the AI, and the AI should support humans in
making more informed decisions. Human oversight, then, is intended to ensure that the AI
does not undermine human autonomy or cause other undesirable efects. [
S49, S59, S63, S70, S90, S98, S100, S103, S108, S110] • Fake news or misinformation [S2, S3, S25, S30, S38, S42, S48, S67, S81, S93, S96, S97,
S101, S103, S107, S110]
Loss of learning or ease of cheating was one of the most commonly discussed ethical issues for LLMs. The discussion in the literature was primarily forward-facing, focusing on future issues. Only three papers [S59, S63, S110] were empirical in nature. To mitigate such issues, literature proposes (a) using teaching approaches that leverage LLMs and other AI tools while making it dificult to use them to cheat, such as in-person assignments, group work, and oral presentations, (b) providing better instructions to make students less inclined to use them, and (c) to better combat cheating with LLMs.
Fake news and other misinformation and disinformation. LLMs can be utilized to generate vast amounts content, and LLM-generated content is already not easily identifiable by humans and only becoming more dificult to discern. As a mitigation measure, literature recommends AI solutions for detecting AI-generated content, but this is acknowledged as a moving goalpost.
Echo chambers in LLMs manifest in that LLMs are not prone to challenging the views of their users and typically passively agree with the user, providing the content requested, barring any guardrails.
Self-acting AI refers to LLMs making decisions without suficient human oversight, ranging from issues such as anthropomorphic LLMs expressing human-like emotions (e.g., ”I’m sorry to hear that”) to unobserved harmful actions.
Influence through suggestions. AI outputs are observed to afect user attitudes towards concepts [S38], and influence the entire creative process even if not accepted into the final text directly [S66]. Concerns have also been raised over AI afecting users’ self-image [S8].
In terms of manipulation, literature discusses both purposeful manipulation and ”nudging”, as well as unintended manipulation resulting from biased training data. The black box nature of LLMs and the lack of alternative answers unless actively prompted are also considered to negatively impact human agency [S102]. As mitigation measures, literature suggests human-inthe-loop, clearly disclosing AI-generated content, and increased regulation.
This requirement, on a general level, posits that AI systems should be designed in a way that
minimizes any harm they may cause, while preventing unacceptable harm. This includes
cybersecurity issues, accuracy of the models being used, having fallback plans in case of issues,
as well as reliability and reproducibility [
Inaccurate results of LLMs are a widely discussed issue in literature. Examples include attributing nationalities solely based on names [S112] and repeating common misconceptions found online [S107]. This is particular a problem in LLMs as humans are inclined to look at given content as a whole rather than zooming in on details, and LLMs are very capable of producing content that overall looks professional [S29]. Inaccuracies also tend to increase with model size [S107]. Better training data is generally suggested to mitigate these issues, whereas current models tend to use erratic data scoured from all over the Internet.
Dangerous content refers to potential for physical harm, such as dangerous advice. This includes both overtly unsafe (e.g., ”drink poison”) and indirectly unsafe content (”eat a Carolina Reaper”) [S76]. Children are considered most vulnerable in this regard, but adults can also fall victim to, e.g., bad legal or home appliance maintenance advice [S97].
Alignment refers to AI working in alignment with ethical values humans wish to impose on it, and is discussed in technical papers as well. In this SLR, we associated papers discussing unintentionally produced unethical content to relate to alignment. Such unintentional nonalignment is generally attributed issues with training data.
Bias scoring and solutions were discussed as a problem rather than a solution in many papers, as creating bias-free bias measures is a challenge in and of itself [S21, S67, S116]. Models may be trained to score well on a specific metric as a form of ”fairwashing” [S21]. In the context of LLMs, new issues may arise, as, e.g., filtering out certain words without context can eliminate reclaimed slur-words and removal of discourse of minorities. As mitigation measures, literature calls for standardized and validated bias measurement methods, as well as new types of metrics.
Data leakage or unintended memorization refers to LLMs memorizing their training data verbatim and occasionally outputting direct excerpts of it, which is considered a safety and robustness concern here as much as it is a data privacy concern. This is an issue especially for private data, which may also be purposefully extracted by malicious actors as opposed to simply being accidentally spilled. Several studies empirically demonstrate data leakage in existing LLMs [S40, S87, S88, S89]. Existing defensive techniques are criticized [S55, S88, S89], with the scale of LLMs working against them due to the equally large attack space it provides. Memorization is observed particularly for unique or rare data items, and it seems to increase with model capacity [S41]. As mitigation measures, literature proposes techniques for reducing memorization (e.g., [S41, S68, S87, S115]) and overall calls for improved techniques and designs to combat it.
This requirement focuses on issues related to data and privacy in the development of AI systems. It encompasses both input data as well as system outputs. Current ML systems are particularly data intensive, and LLMs even moreso, emphasizing the importance of ethical issues related to data. The following issues related to this principle were identified from the literature:
Data gathered without consent is a much-discussed issue in relation to LLMs due to current LLMs being typically trained on massive sets of data collected from all over the Internet. E.g., private data that is publicly available online may have ended up there as a result of a data breach rather than being uploaded with consent [S44], and utilizing it may cause harm [S81]. As the conceptually but practically challenging simple solution, literature recommends the use of data sets where data is gathered with consent [S44].
Privacy and data security is a widely discussed issue in LLMs, as it is with AI overall. In LLMs, aside from the training data posing various issues, the data collected from the users is also a potential issue [S9, S110]. Children in particular may be prone to sharing personal information with LLMs while interacting with them, although this is an issue for adults as well, especially with anthropomorphized AI [S97]. Various mitigation measures are discussed in literature, though each with their practical problems: automated de-identification [S47], pseudonymization and other de-identification methods [S11, S45, S47, S56, S97], and synthetic data [S47], as well as humans-in-the-loop [S50] and better sourcing of training data [S44, S97].
Data management poses challenges for LLM developers due to the massive amounts of training data used by current LLMs, making traditional data management methods non-efective [S111], posing both ethical and legal data management issues [S39]. As a mitigation measure, literature calls for increased governance [S39, S97] (although existing approaches are seen as insuficient), more auditable models [S56], and better documentation of training data sets [S71].
Transparency is focused on making AI applications more understandable to various
stakeholders, including their users, through more specific aspects such as traceability, explainability,
and communication [
A large number of papers raise concerns related to academic integrity and source tracking in the context of LLMs, as an increasing number of research papers (or manuscripts) is generated with the aid of LLMs or entirely by LLMs. Publishers have expressed concerns over their open access publications being used to train LLMs [S24]. As mitigation measures, literature again calls for authenticity checks for identifying AI-generated content [S9, S20, S24, S43, S44, S60, S62, S100], although false positives need to be seriously considered given the ramifications for researchers [S62]. Additionally, literature stresses the importance of guidelines and rules for using LLMs [S6, S60, S63, S98, S100, S110].
Unfair decision-making, here, refers to improper results that cannot be easily traced. Many of these issues are related to issues with training data, or the goals of the model training process. For example, when prioritizing who receives healthcare first, a value judgement is made [S12], intentionally or unintentionally. Allocational harms are more often considered unintentional harms [S58, S69, S81] and stem from training data that contains biases.
Lack of transparency is an issue in current LLMs that are predominantly black boxes, resulting in various issues. LLM users may create so-called ”algorithmic folk theories” where they attribute models with too much authority and fail to understand their limitations [S66]. This lack of transparency also increases reliance on the largest LLM developers [S101] and hinders research on LLMs. Moreover, current LLMs generally sufer from documentation debt [S81]. Better documentation and communication are proposed as mitigation measures.
Copyright infringement, here, refers to creative ownership rather than data ownership (discussed elsewhere). Current LLMs do not indicate whose creations were used to produce which outputs. LLMs do not credit creators and are themselves incapable of considering legal or ethical issues [S2]. No mitigation measures are proposed due to the technical limitations of current LLMs, and relevant legislation is still pending.
Unfactual training data refers to factually incorrect training data, which is particularly an issue for LLMs due to the vast amounts of (poorly curated) training data used to train them. This is also a challenge as new discoveries are constantly made in science, occasionally rendering obsolete information that was used to be considered a fact. Current LLMs are often trained on static data and not actively updated.
A trustworthy AI application needs to be fair and respect all people in an equal manner [
Biased training data or outputs were the most common issue discussed in the literature reviewed for this SLR. The 10 most commonly discussed types of bias are biases related to gender, age, sexual orientation, physical appearance, disability, ethnicity, socioeconomic, religion, and culture, as well as cross-sectional bias [S16, S105]. More conceptual types of bias are discussed in [S51]: confidence, recency, majority label, and common token bias. As mitigation measures, literature proposes better training data management (both manual annotation and technological innovation) [S16, S35, S42, S46, S72, S105], pre-curated training corpora [S93, S94, S113], and making the NLP community more diverse [S16, S42].
Discriminatory results were another widely discussed issue in the reviewed literature, which focused on gender discrimination (e.g., [S46, S72]) and discrimination against marginalized demographic groups (e.g., [S58, S84]). This included a number of papers presenting empirical results [S46, S72, S73, S77, S80, S84, S109]. For mitigation measures, literature proposes multicultural development teams [S58], continuous evaluation [S77], and adversarial triggers and prompt engineering for testing purposes [S77, S83].
The lack of a global definition for bias and fairness presents issues for LLMs and AI systems at large. Currently, ethics reflected in the development of LLMs are those of western, white populations [S9, S116]. This also relates to the larger discussion on the limitations of algorithmic fairness. Few mitigation measures are proposed.
Non-binary gender is neglected often in LLMs due to the constraints of language, it being easier to study a model with a binary designation of gender. As bias specifically in LLMs is a new practical issue overall, few solutions are currently discussed in literature to this more specific bias issue
Toxic content illustrates the problems of extracting training data from the Internet with little curation. Data used to train LLMs often includes, e.g., content from Reddit that is from banned or quarantined subreddits [S54]. Many of the reviewed studies discussing this issue also demonstrate it in practice. As for mitigation measures, though toxicity scoring tools similar to bias scoring ones exist, they have similar problems [S22], and using smaller, curated sets of data hinders performance.
It is argued that LLMs promote inequality in diferent ways. E.g., by ofering better performance against subscription fees [S7], as a result of those with power producing most content used to train them [S113], and by exhibiting poorer performance in languages other than English [S67, S91, S113]. Mitigation measures are shared between other bias and fairness issues.
According to AI HLEG [
Loss of social skills may occur in the future if increased use of AI results in decreased development of social skills. As this is a hypothetical future issue, discussed to raise awareness of its potential future impacts, no mitigation methods are presented to tackle it currently.
LLMs may reduce the value of education if students become too reliant on LLM use and feel that learning and memorizing some things themselves is not worth the efort. Using LLMs to pass tests may also result in students being ill-prepared for future careers [S29]. Literature suggests teaching AI literacy as one way of addressing such issues [S108].
Paper or credential generation. LLMs are capable of producing superficially viable research content, which can lead to an inflated number of papers being produced. E.g., some individuals may use LLMs to produce massive amounts of self-referential studies with little scientific novelty [S20, S24]. As mitigation measures, literature proposes guidelines and demanding disclosure of LLM use [S35] and technologies for identifying LLM-generated content.
Aside from unintentional unethical content, LLMs can be used to produce purposeful toxic or immoral content. E.g., to generate malware [S92], or for social engineering attacks [S92, S99]. Adversarial triggers can be used to generate unexpected outputs seemingly unprompted [S99]. Mitigation measures proposed are similar to those proposed for toxic and biased content overall, in addition to stock responses for inappropriate prompts [S91].
Like many new technologies before, LLMs may result in job loss or class divide. In particular, LLMs are seen to concern groups of professionals that have historically not been under such threats before, namely white-collar workers [S3] and creative professionals [S37, S97].
Training data pruning for LLMs is currently often conducted by human actors from developing countries [S2]. These individuals are subjected to large amounts of unwanted and inappropriate content in the process, which can be extreme and emotionally damaging.
Environmental impacts of LLMs include (1) direct impacts from energy use, (2) secondary impacts from emissions, (3) impacts resulting from the system influencing human behavior, and (4) resources needed for hardware. Few operators publish figures on environmental impacts [S106], and as the scale of LLMs continues to grow, they require more and more computational power [S106]. Notably, increasing model fairness tends to decrease sustainability as well [S17]. The following mitigation measures are proposed: increased (and standardized) reporting [S17, S71, S81], more eficient solutions [S17, S32], running systems in more carbon friendly regions [S81].
Fairwashing (cf. greenwashing) is discussed in literature in relation to hypothetical certificates that may be granted to systems and their developers without reliable means of verifying compliance to the methods the certificate aims to measure, as well as overall marketing of LLMs and AI.
Replacement of traditional learning, while closely related to reduced value of education, which in this SLR refers to similar issues from the perspective of students, is more focused on the entire education system. Thus, it includes other actors than learners as well. E.g., for educators, the use of LLMs in and of itself warrants ethical consideration when it comes to tasks such as (automated) grading [S53, S57]. Mitigation measures include AI literacy education from an early age [S2].
The seventh and final requirement of the AI HLEG [
Corporate influence is seen as an issue in relation to LLMs as large, multinational corporations dominate the LLM development and consumer market, and do so largely unregulated at present. These corporations safeguard their position by obscuring the training data and program code [S13] (and own the large cloud computing facilities for training), and control research by publications and recruitment [S101, S116]. Smaller practitioners struggle to enter the market. As mitigation measures, literature calls for more regulation [S2, S10, S23, S25, S96] and better funding for independent researchers [S101, S116].
Cost of privacy is an issue in LLMs in that it can often be dialed down to improve eficiency or profitability. The massive amounts of training data make comprehensive human overview of the training material for LLMs practically impossible [S56, S89]. Due to it being so expensive to improve model privacy, forgoing it may be tempting, especially if the model is for internal organization use [S45], but organizations should remember that such models should not be shared or used in situations where this may be more likely to cause issues.
As for the cost of AI monitoring, continuously monitoring, testing, and amending LLM performance after deployment can be highly resource-intensive. In particular, re-training models is especially expensive in the context of LLMs, which may be required to address ethical issues when there is a will to do so.
Ambiguity of accountability presents various issues for LLMs. The fact that LLMs do not clearly emulate any parts of their training data also has implications for copyright and academic referencing, and may also be relevant for toxic and otherwise inappropriate content [S13]. E.g., if an LLM makes a social media post that breaks the terms of service or is illegal, pinning the responsibility on a human actor may be dificult, if it is needed. As a solution, literature suggests enforcing accountability through regulation.
The SLR presented in this paper has its limitations. The SLR protocol itself has the following
limitations. First, we have limited the search to literature published after 2020. This was done
due to much of the LLM-specific discussion being related to very recent advances in the field
(and, e.g., the introduction of ChatGPT), so as to more reliably and easily exclude irrelevant
results. However, this will have also excluded any less recent papers with the foresight to discuss
these issues prior to 2021. Second, we have excluded some of the requirements (robustness,
safety, governance, societal, environmental, and accountability) of the AI HLEG [
Additionally, it is also prudent to acknowledge potential limitations related to the data
extraction and analysis. The analysis process was carried out by the first author, although
the second and third author discussed and planned the process with the first author. This
leaves room for subjective interpretation on part of the first author in terms of categorizing the
identified ethical issues. However, we argue that the use of an existing ethical framework (AI
HLEG [
In this paper, we conducted an SLR of literature on ethical issues and risks associated with
LLMs and generative AI. Based on 116 papers, we identified 434 individual ethical issues, based
on which we proposed 39 categories of ethical issues related to LLMs. We then mapped these
39 issue categories to the seven requirements for ethical AI found in the Ethics Guidelines
for Trustworthy AI [
First, we conducted our SLR using AI HLEG [
However, when looking at each issue in more detail, some still present new practical
challenges, while others are indeed familiar ones. As an example, the trade-of between accuracy
and explainability (or interpretability) is well-documented in existing literature (e.g., [
Secondly, based on the literature, it seems that there is more emphasis on user responsibility as
well, in addition to developer or organization responsibility that has typically been the focus in
AI ethics (e.g., as discussed in AI HLEG [
Thirdly, our principle-based SLR approach compliments study of Weidinger et al. [
Finally, as for practical implications, the issues identified in this SLR may inform any interested practitioner in potential issues and risks they may want to be aware of depending on their project or system context. In addition to identifying potential ethical issues in this SLR, we have also listed the solution suggestions for each ethical issue found in literature, which may help practitioners tackle any issues they may consider relevant for their system context.
During the preparation of this work, the author(s) used ChatGPT for Sections 3.1 and 3.2 (SLR protocol): Drafting content. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.
This appendix contains the full list of studies included in this SLR (Table 2), clearly separated from the references used outside Section 4 that were not a part of the SLR itself. As mentioned in Section 3, these papers were each given an ID (S1-S116) that was used to reference them in the paper (e.g., ”[S52]”).
S52 Cooper, G. (2023). Examining science education in ChatGPT: An exploratory study of generative artificial intelligence. Journal of Science Education and Technology, 32(3), 444-452.
S53 Poulton, A., & Eliens, S. (2021, September). Explaining transformer-based models for automatic short answer grading. In Proceedings of the 5th International Conference on Digital Technology in Education (pp. 110-116).
S54 Nguyen, H., Malik, A., & Zink, M. (2022). Exploring Realtime Conversational Virtual
Characters. SMPTE Motion Imaging Journal, 131(3), 25-34.
S55 He, X., Chen, C., Lyu, L., & Xu, Q. (2022). Extracted BERT Model Leaks More Information than You Think!. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 1530–1537), Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
S56 Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., ... & Rafel, C. (2021). Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) (pp. 2633-2650).
S57 Kumar, R. (2023). Faculty members’ use of artificial intelligence to grade student papers: a case of implications. International Journal for Educational Integrity, 19(1), 9. S58 Ramesh, K., Sitaram, S., & Choudhury, M. (2023). Fairness in language models beyond English: Gaps and challenge. In Findings of the Association for Computational Linguistics: EACL 2023 (pp. 2106–2119), Dubrovnik, Croatia. Association for Computational Linguistics.
S59 Biderman, S., & Raf, E. (2022, October). Fooling MOSS detection with pretrained language models. In Proceedings of the 31st ACM international conference on information & knowledge management (pp. 2933-2943).
S60 Dergaa, I., Chamari, K., Zmijewski, P., & Saad, H. B. (2023). From human writing to artificial intelligence generated text: examining the prospects and potential threats of ChatGPT in academic writing. Biology of sport, 40(2), 615-622.
S61 Li, Y., Zhang, G., Yang, B., Lin, C., Wang, S., Ragni, A., & Fu, J. (2022). Herb: Measuring hierarchical regional bias in pre-trained language models. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022 (pp. 334–346). Association for Computational Linguistics.
S62 Wahle, J. P., Ruas, T., Kirstein, F., & Gipp, B. (2022). How large language models are transforming machine-paraphrased plagiarism. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 952–963). Association for Computational Linguistics.
S63 Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28(11), 13943-13967.
S64 Kasirzadeh, A., & Gabriel, I. (2023). In conversation with artificial intelligence: aligning language models with human values. Philosophy & Technology, 36(2), 27.
S65 Pikuliak, M., Beňová, I., & Bachratý, V. (2023). In-depth look at word filling societal bias measures. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (pp. 3648–3665), Dubrovnik, Croatia. Association for Computational Linguistics.
S66 Bhat, A., Agashe, S., Oberoi, P., Mohile, N., Jangir, R., & Joshi, A. (2023, March). Interacting with next-phrase suggestions: How suggestion systems aid and influence the cognitive processes of writing. In Proceedings of the 28th International Conference on Intelligent User Interfaces (pp. 436-452).
S67 Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2022). Language generation models can cause harm: So what can we do about it? an actionable survey. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics (pp. 3299–3321), Dubrovnik, Croatia. Association for Computational Linguistics.
S68 Li, X., Tramer, F., Liang, P., & Hashimoto, T. (2022). Large language models can be strong diferentially private learners. In Proceedings of the 10th International Conference on Learning Representations (ICLR 2022).
S69 Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022). Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3), 258-268.
S70 Crawford, J., Cowling, M., & Allen, K. A. (2023). Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). Journal of University Teaching & Learning Practice, 20(3), 02.
S71 Alshahrani, S., Wali, E., & Matthews, J. (2022, December). Learning From Arabic Corpora But Not Always From Arabic Speakers: A Case Study of the Arabic Wikipedia Editions. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP) (pp. 361-371).
S72 Borchers, C., Gala, D. S., Gilburt, B., Oravkin, E., Bounsi, W., Asano, Y. M., & Kirk, H.
R. (2022). Looking for a handsome carpenter! debiasing GPT-3 job advertisements. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP) (pp. 212–224), Seattle, Washington. Association for Computational Linguistics. S73 Kraft, A., Zorn, H. P., Fecht, P., Simon, J., Biemann, C., & Usbeck, R. (2022). Measuring gender bias in german language generation. In Proceedings of The Informatik 2022 Workshop ”Trustworthy AI in Science and Society” (pp. 1257-1274), Hamburg. 26.-30.
September 2022.
S74 Touileb, S., & Nozza, D. (2022). Measuring harmful representations in Scandinavian language models. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) (pp. 118–125), Abu Dhabi, UAE. Association for Computational Linguistics.
S75 Nozza, D., Bianchi, F., Lauscher, A., & Hovy, D. (2022). Measuring harmful sentence completion in language models for LGBTQIA+ individuals. In Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion. Association for Computational Linguistics. S77 S78 S79 S80 S81 S82 S83 S84 S85 S86
Mei, A., Kabir, A., Levy, S., Subbiah, M., Allaway, E., Judge, J., ... & Wang, W. Y. (2022). Mitigating covertly unsafe text within natural language systems. In Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 2914–2926), Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Venkit, P. N., Gautam, S., Panchanadikar, R., Huang, T. H. K., & Wilson, S. (2023). Nationality bias in text generation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 116–122, Dubrovnik, Croatia. Association for Computational Linguistics.
Longoni, C., Fradkin, A., Cian, L., & Pennycook, G. (2022, June). News from generative artificial intelligence is believed less. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (pp. 97-106).
Dev, S., Sheng, E., Zhao, J., Amstutz, A., Sun, J., Hou, Y., ... & Chang, K. W. (2021). On measures of biases and harms in NLP. Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022 (pp. 246–267). Association for Computational Linguistics.
Akyürek, A. F., Paik, S., Kocyigit, M. Y., Akbiyik, S., Runyun, Ş. L., & Wijaya, D. (2022). On measuring social biases in prompt-based multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022 (pp. 551–564), Seattle, United States. Association for Computational Linguistics.
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency (pp. 610-623). Vashishtha, A., Prasad, S. S., Bajaj, P., Chaudhary, V., Cook, K., Dandapat, S., ... & Choudhury, M. (2023, May). Performance and Risk Trade-ofs for Multi-word Text Prediction at Scale. In Findings of the Association for Computational Linguistics: EACL 2023 (pp. 2226-2242).
Abid, A., Farooqi, M., & Zou, J. (2021, July). Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (pp. 298-306).
Qian, R., Ross, C., Fernandes, J., Smith, E., Kiela, D., & Williams, A. (2022). Perturbation augmentation for fairer NLP. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 9496–9521), Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Nozza, D., Bianchi, F., & Hovy, D. (2022). Pipelines for social bias testing of large language models. In Proceedings of BigScience Episode# 5–Workshop on Challenges & Perspectives in Creating Large Language Models. Association for Computational Linguistics.
Elmahdy, A., Inan, H. A., & Sim, R. (2022). Privacy leakage in text classification: A data extraction approach. In Proceedings of the Fourth Workshop on Privacy in Natural Language Processing (pp. 13–20), Seattle, United States. Association for Computational Linguistics.