Digital labor platforms have evolved and diversified under the influence of artificial intelligence (AI) technology over the last couple of years due to the multimodal transformative capabilities of large language models (LLMs) and generative agent-based models. These platforms are now established and offer scalable solutions to solve macrotasks of varied levels of complexity and demands. However, the challenges associated with the inappropriate use of AI in digital labor settings are enormous. This emphasizes the need of collaboration mechanisms enabling crowds, large groups, or self-organizing teams to create new solutions or just responsibly oversee AI-generated outputs. Despite growing scholarly interest in macrotask-based digital labor platforms, there remains a significant gap in understanding how AI-augmented collaboration can shape the socio-technical dynamics of the digital economy. This paper contributes to this stream of research by providing a new lens on the potential threats, enablers, and open questions at the intersection of human-centered AI and large-scale collaboration in digital labor platforms with crowdsourcing at the heart of it.
Digital labor platforms have become well-established and widely used in a wide range of
problems encountered daily by companies and institutions worldwide. Online workers
operating remotely in real-time or asynchronous modes can be effective in contexts that involve
product feature development, interface design, transcription, film production, etc. Traditionally,
digital labor platforms have primarily supported microtask-based models, where online workers
perform decontextualized tasks that can later be aggregated by requesters [
As the digital economy evolves towards more knowledge-intensive and creative tasks, the
role of artificial intelligence (AI)-augmented collaboration has gained increasing attention.
Collaboration offers a promising path forward for optimizing processes and overcoming
workflow fragmentation. To support this transition, platforms must provide mechanisms
beyond basic coordination by facilitating coalition-based ensembles where contributors build on
each other’s work and make joint decisions mediated by AI [
Extensive field experiments have been carried out to demonstrate the transformative effect
of AI-augmented collaboration. In domains such as tissue image annotation, for instance,
human annotators working alongside AI-driven preprocessing tools have achieved improved
accuracy and efficiency [
This paper seeks to provide a descriptive account of the key challenges, potential directions, and existing gaps in AI-augmented collaboration for crowdsourcing applications. We focus specifically on the potential of designing AI systems that enhance collaboration in macrotask environments. To this end, we continue our effort to consolidate research and practical insights outlining foundational pathways for further investigation and technological advancement in this area of work.
The rise of digital labor platforms has transformed the work landscape by offering new ways to
outsource tasks that are either expensive or too complex to automate. As these platforms gain
traction as viable solutions for addressing economically infeasible tasks through traditional
means, researchers have turned their attention to the collective intelligence that emerges from
coordinated crowd activity mediated by AI [
Despite the growing academic interest in AI-augmented collaboration, many digital labor
platforms lack built-in support for communication or coordination among workers who use
external tools such as social forums or mobile messaging apps to exchange knowledge and
provide information about ongoing tasks [
Recent initiatives have begun to address such limitations through intelligent systems and
algorithms that facilitate the dynamic assembly and coordination of groups or teams of online
workers. Here, the role of AI is not merely to mediate the workflow but a key enabler of
enhanced collaboration able to actively augment human abilities through informed
decisionmaking support or real-time feedback. Advanced task assignment algorithms now factor in
social affinity, worker compatibility, and motivational incentives [
The convergence of AI and collaborative crowdsourcing introduces critical challenges and research opportunities. From a design perspective, questions of transparency, trust, and fairness become central as AI agents take on greater roles in mediating human labor. Equally important is the need to examine the multidisciplinary nature of human-AI mixed-initiative systems. This intersection forms the basis for a research agenda intended to unpack the socio-technical elements that characterize AI-augmented collaboration in digital labor platforms.
From mixed-initiative evaluation to chat-based anonymous communication, generative AI and LLM-based multimodal tools now support an increasingly diverse range of interactive features. As illustrated in Figure 2, certain aspects must be considered when integrating AI into digital labor platforms. In this section, we list some of the opportunities, threats, and prominent areas of application of AI-augmented collaboration in crowdsourcing settings.
AI-augmented collaboration offers significant potential for optimizing both collective output
and the workflows associated with task assignment in crowdsourcing. Traditional models of
crowd work have often relied on simplistic assumptions about worker capabilities and random
or rule-based task distribution. However, AI-based methods enable more nuanced and adaptive
strategies matching tasks to individuals based on multifactorial models. These models account
for a range of attributes, including worker skills, preferences, motivation, and historical
performance indicators such as accuracy and response time. Such personalization has been
shown to enhance output quality while also reducing task completion times. This approach was
discussed by Retelny and co-authors [
A persistent challenge in open-ended crowdsourcing environments lies in managing quality
control amidst the inherent variability of crowd workers’ reliability and expertise [
In the context of crowdsourcing challenges on business ideation, hybrid teams using an iterative
prompting strategy (where humans guided the LLM to explore diverse solution directions)
outperformed both independent human teams and unguided AI, demonstrating the potential of
combining human strategic guidance with AI’s generative capacity [21]. As LLMs become
increasingly integrated into crowdsourcing workflows, the boundary between human- and
AIgenerated contributions has blurred, raising critical concerns about trust, authorship, and the
epistemic validity of collected data [
The execution of human intelligence tasks (HITs) with the computational support of AI systems underscores the need for a more ethically aware, diverse, skilled, and AI-literate crowd workforce. There is growing recognition that the unregulated use of LLMs may lead to homogenized outputs, which undermines the goal of many crowdsourcing initiatives aimed at capturing heterogeneity in attitudes, behaviors, and lived experiences. In response, hybrid frameworks have been proposed and tested in complex tasks such as misinformation detection, content moderation, and deepfake identification [12]. However, further research is required to develop effective design principles for human-LLM interaction within digital labor contexts. One possible avenue is investigating optimal strategies for task decomposition between humans and AI agents based on user interface (UI) designs that clearly delineate AI contributions and facilitate human oversight and correction.
Given its importance, the aggregation and interpretation of results generated by both human and non-human agents present novel methodological challenges. As pointed out in [13], the success of AI-crowd interactive systems depends on the development of robust mechanisms to reconcile outputs from mixed-agent teams and to ensure that the collective intelligence produced remains trustworthy, diverse, and aligned with the task’s epistemic goals. This entails exploring new aggregation techniques that can account for the different levels of human and AI influence on the generated data, which could be leveraged through differential weighting or qualitative analysis of contributions. In the context of AI recommendations, if the AI’s reasoning or criteria are opaque, users tend to distrust its outputs or feel that the process is unfair. Providing explanations for AI recommendations can increase user trust and achieve more accurate results [22], despite recent evidence indicating that this may be insufficient to ensure critical evaluation and appropriate incorporation of human and AI contributions in decisionmaking [34].
The integration of LLMs into collaborative settings also raises important questions about the skill sets required from workers. Although AI is expected to automate routine tasks and enable crowd workers to engage in higher-level cognitive activities, the growing use of generative AI in these contexts often diminishes perceived work value and enjoyment while simultaneously introducing new ethical concerns [23]. This underscores the need for training and upskilling initiatives to equip the collaborative workforce for new roles in AI-augmented environments and to mitigate potential inequalities among workers [24]. Moreover, the legal and ethical implications of authorship and intellectual property underlying content co-created with LLMs require careful consideration. A recent example is the viral spread of Studio Ghibli-themed, prompt-generated animations which have raised concerns regarding ownership and privacy. Therefore, guidelines and frameworks are needed to address issues of accountability, opacity, responsibility, and fair attribution of contributions in AI-enabled collaborative crowdsourcing. To create and implement such policies, it is important to bring experts from different fields to define ethical principles that can contribute to avoid the misguided and misunderstood usage of AI. Digital labor platforms should then implement these principles and regulations in their terms of service, offering tools to track contributions and resolve disputes by establishing effective social conventions among humans and LLM populations [38]. Among the vast amount of possibilities, potential solutions include providing clear usage policies, ethical instructions, and training to enhance workers’ AI literacy.
Introducing LLMs into digital labor platforms has amplified existing power imbalances. Algorithmic biases are usually rooted in opaque model training and therefore amplify social injustices and inequalities [14, 35]. These dynamics are particularly concerning in crowdsourcing contexts where the labor force is often diverse but socioeconomically precarious. To mitigate this, inclusivity must be embedded within AI-augmented systems [15]. Preserving worker autonomy and ensuring equitable task allocation across demographic and cultural lines are essential steps toward fostering a fairer digital labor ecosystem [27]. On top of all of this, the lack of transparency in how AI algorithms evaluate workers’ performance can exacerbate feelings of injustice and hinder opportunities for skill development and social inclusion on these platforms [28, 29]. It is thus paramount to explore explainable AI and other related strategies within collaborative settings. Also, it is necessary to develop mechanisms as proposed by Tubella and co-authors [25] to understand the behavioral constraints of the AI system and how they influence its outputs.
AI-driven collaborative crowdsourcing offers a potential way of addressing structural imbalances inherent in digital labor markets. These markets often operate asymmetrically, concentrating power in the hands of platform owners and requesters. This may lead to systemic inequities in how tasks are assigned and assessed [16]. As mentioned by Colón Vargas [26], the operation of the AI industry is often characterized by intellectual appropriation and extreme exploitation of workers (e.g., data labelers) from minority workforces. This asymmetry undermines worker wellbeing, limits agency, and contributes to an ongoing sense of precarity. In rapidly evolving domains like user experience (UX) design, AI systems that learn from usergenerated data may inadvertently replicate and reinforce these problems over time [17].
Addressing these issues demands a paradigm shift toward more ethical, culturally sensitive, and worker-centered design in both academic and industrial AI research. This involves a deeper exploration of alternative platform governance models able to empower workers by providing them greater agency over platform policies and operations. Furthermore, it is crucial to consider mechanisms that facilitate collective negotiation for crowd workers, enabling them to advocate for their rights and interests more effectively. In addition to governance, developing fair compensation models is essential. As noted in [33], such models should move beyond metrics that exclusively prioritize speed or work volume. Instead, they should account for the inherent complexity of tasks and the actual value of the contributions made by individual workers. By moving towards a more nuanced compensation model, we can help establish a digital labor ecosystem that is more equitable in the long term, ensuring fairer rewards for the expertise and effort involved.
Despite the growing reliance on AI in crowdsourcing, collaboration has historically received limited attention in the literature [18]. As the field progresses, a more humanized approach to crowd work is needed, one that recognizes workers not merely as task solvers but as active agents with different cultural contexts, learning trajectories, and social motivations. Socioalgorithmic approaches offer a promising path forward by enabling adaptive personalization of tasks, feedback mechanisms, and social incentives in ways that align with each worker’s skills and characteristics. Additionally, crowd workers can play a more participatory role within AI auditing by helping to detect bias, uncover model vulnerabilities, and correct hallucinations [33]. This approach repositions crowd workers as co-creators and “stewards” of ethical AI systems toward empowerment and shared responsibility. To advance this humanistic approach of crowd work, more research should be conducted on how AI-mediated communication affects online community formation within these platforms. Understanding how AI can facilitate trustbuilding and mutual support among workers is critical for fostering a more equitable and sustainable digital labor ecosystem. As AI evolves, design approaches that reduce algorithmic aversion by aligning system functionality with users’ expectations must be prioritized as a way of ensuring an inclusive and effective crowd-AI digital workforce.
This paper critically examines the role of AI-augmented collaboration in addressing both decomposable and non-decomposable macrotasks within crowdsourcing environments. Our agenda opens up new avenues for inquiry by highlighting the threats underlying workers’ engagement with both harmful and beneficial AI-generated content. Moreover, the paper explores the socio-technical arrangements and inequalities faced by workers in relation to the integration of AI into digital labor platforms. However, our work still faces significant limitations in clarifying how AI affects the labor conditions of crowd workers, including wages and income distribution. Addressing this requires more case studies and concrete examples of how digital labor platforms are being used to train LLMs. Furthermore, data is still lacking to contextualize the AI-augmented collaborative crowdsourcing phenomenon and its relevance and distribution across countries, sectors, and occupations.
The authors have not employed any Generative AI tools. [12] X. Zeng, D. La Barbera, K. Roitero, A. Zubiaga, S. Mizzaro, Combining large language models and crowdsourcing for hybrid human-AI misinformation detection, in: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 2332–2336. [13] T. Tamura, H. Ito, S. Oyama, A. Morishima, Influence of AI’s uncertainty in the DawidSkene aggregation for human-AI crowdsourcing, in: Proceedings of the International Conference on Information, 2024, pp. 232–247. [14] E. Jussupow, M. A. Meza Martínez, A. Maedche, A. Heinzl, Is this system biased? – How users react to gender bias in an explainable AI system, in: Proceedings of the International Conference on Information Systems, 2021, 11. [15] J. Y. Jung, S. Qiu, A. Bozzon, U. Gadiraju, Great chain of agents: The role of metaphorical representation of agents in conversational crowdsourcing, in: Proceedings of the CHI Conference on Human Factors in Computing Systems, 2022, pp. 1–22. [16] K. Hansson, T. Ludwig, Crowd dynamics: Conflicts, contradictions, and community in crowdsourcing, Computer Supported Cooperative Work, vol. 28, 2019, pp. 791–794. [17] J. Kay, A. Kasirzadeh, S. Mohamed, Epistemic injustice in generative AI, in: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2024, pp. 684–697. [18] D. F. Donglai, L. Yanhua, Trust-aware task allocation in collaborative crowdsourcing model, Computer Journal, vol. 64, no. 6, 2021, pp. 929–940. [19] R. Chaves, C. E. Barbosa, G. A. de Oliveira, A. Lyra, M. Argôlo, H. Salazar, Y. Lima, D.
Schneider, A. Correia, J. M. de Souza, Charting a course at the human–AI frontier: A paradigm matrix informed by social sciences and humanities, AI & SOCIETY, 2025, pp. 1– 14. [20] S. Noy, W. Zhang, Experimental evidence on the productivity effects of generative artificial intelligence, Science, vol. 381, no. 6654, 2023, pp. 187–192. [21] L. Boussioux, J. N. Lane, M. Zhang, V. Jacimovic, K. R. Lakhani, The crowdless future? Generative AI and creative problem-solving, Organization Science, vol. 35, no. 5, 2024, pp. 1589–1607. [22] M. Vössing, N. Kühl, M. Lind, G. Satzger, Designing transparency for effective human-AI collaboration, Information Systems Frontiers, vol. 24, no. 3, 2022, pp. 877–895. [23] P. Mei, D. N. Brewis, F. Nwaiwu, D. Sumanathilaka, F. Alva-Manchego, J. Demaree-Cotton, If ChatGPT can do it, where is my creativity? Generative AI boosts performance but diminishes experience in creative writing, Computers in Human Behavior: Artificial Humans, vol. 4, 2025, 100140. [24] A. Humlum, E. Vestergaard, The unequal adoption of ChatGPT exacerbates existing inequalities among workers, Proceedings of the National Academy of Sciences, vol. 122, no. 1, 2025, e2414972121. [25] A. A. Tubella, A. Theodorou, V. Dignum, F. Dignum, Governance by glass-box: Implementing transparent moral bounds for AI behaviour, arXiv preprint arXiv:1905.04994, 2019. [26] N. Colón Vargas, Exploiting the margin: How capitalism fuels AI at the expense of minoritized groups, AI and Ethics, vol. 5, 2024, pp. 1871–1876. [27] A. Kittur, J. V. Nickerson, M. S. Bernstein, E. M. Gerber, A. Shaw, J. Zimmerman, M. Lease, J.
J. Horton, The future of crowd work, in: Proceedings of the Conference on Computer Supported Cooperative Work, 2013, pp. 1301–1318. [28] C. Toxtli, S. Suri, S. Savage, Quantifying the invisible labor in crowd work, Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2), 2021, pp. 1–26. [29] T. J.-J. Li, Y. Lu, J. Clark, M. Chen, V. V. Cox, M. Jiang, Y. Yang, T. Kay, D. M. Wood, J.
Brockman, A bottom-up end-user intelligent assistant approach to empower gig workers against AI inequality, in: Proceedings of the Annual Meeting of the Symposium on HumanComputer Interaction for Work, 2022, pp. 1–10. [30] J. Bisbee, J. D. Clinton, C. Dorff, B. Kenkel, J. M. Larson, Synthetic replacements for human survey data? The perils of large language models, Political Analysis, vol. 32, no. 4, 2024, pp. 401–416. [31] B. R. Anderson, J. H. Shah, M. Kreminski, Homogenization effects of large language models on human creative ideation, in: Proceedings of the Conference on Creativity & Cognition, 2024, pp. 413–425. [32] D. Schneider, R. Chaves, A. P. Pimentel, M. A. de Almeida, J. M. de Souza, A. Correia, AImediated collaborative crowdsourcing for social news curation: The case of Acropolis, in: Proceedings of the ACM International Conference on Interactive Media Experiences, 2025, pp. 395–401. [33] D. Schneider, M. A. de Almeida, R. Chaves, B. Fonseca, H. Mohseni, A. Correia, “Is it future or is it past?”: From self-contained microtasks to AI-driven collaborative crowdsourcing, in: Proceedings of the IEEE International Congress on Human-Computer Interaction, Optimization and Robotic Applications, 2025, pp. 1–4. [34] K. Z. Gajos, L. Mamykina, Do people engage cognitively with AI? Impact of AI assistance on incidental learning, in: Proceedings of the ACM Conference on Intelligent User Interfaces, 2022, pp. 794–806. [35] R. Shelby, S. Rismani, K. Henne, A. Moon, N. Rostamzadeh, P. Nicholas, N’M. Yilla-Akbari, J. Gallegos, A. Smart, E. García, G. Virk, Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction, in: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2023, pp. 723–741. [36] A. Correia, S. Jameel, H. Paredes, B. Fonseca, D. Schneider, Hybrid machine-crowd interaction for handling complexity: Steps toward a scaffolding design framework, in: Macrotask Crowdsourcing: Engaging the Crowds to Address Complex Problems, 2019, pp. 149–161. [37] E. Christoforou, G. Demartini, J. Otterbacher, Crowdsourcing or AI sourcing?,
Communications of the ACM, vol. 68, no. 4, 2025, pp. 24–27. [38] A. F. Ashery, L. M. Aiello, A. Baronchelli, Emergent social conventions and collective bias in LLM populations, Science Advances, vol. 11, no. 20, 2025, eadu9368.