Do Interpersonal Skills Afect Human-AI Collaboration Performance? A Study with ChatGPT

Abstract

Collaboration between humans and artificial intelligence (AI) has demonstrated the potential to achieve performance surpassing that of AI alone. As AI becomes more integrated into society, human-AI collaboration is expected to emerge as a new form of teamwork. The recent advancements in large language models (LLMs) have accelerated research on human-LLM collaboration across various domains. While previous studies have focused on improving the performance of LLMs and methods for efective collaboration, little is known about how user-specific traits, such as interpersonal skills, influence collaboration outcomes with LLMs. This study addresses this gap by focusing on the role of interpersonal skills in human-AI interaction to deepen understanding of human-AI collaboration. The experimental results showed that participants with lower interpersonal skills were more likely to accept AI-generated responses, suggesting that they benefit more from AI. These findings suggest that interpersonal skills could influence how users critically assess with AI-generated content.

-

A

1. Introduction

Collaboration between humans and artificial intelligence (AI) has attracted attention because of its potential to achieve outcomes beyond that of AI alone. This phenomenon, known as the “centaur phenomenon,” originates from a freestyle chess tournament in which a human-AI team named “Centaur” outperformed an AI alone [ 1, 2 ]. This suggests that human intervention plays a crucial role in the efective use of AI. Recently, the collaboration between humans and large language models (LLMs) has garnered growing interest and is being actively explored in writing [ 3, 4 ], education [ 5, 6 ], and various other fields [ 7, 8, 9 ].

However, synergy in human-AI collaboration is not always observed, as it is influenced by the type of task and psychological factors [ 10, 11, 12 ]. To address this issue, researchers are examining the potential for human-AI collaboration from various perspectives such as complementarity [13, 14], trust [15, 16], and teamwork [17, 18]. In addition, researchers are exploring approaches to control LLM strategies for collaborating with humans and other LLMs, informed by psychology and cognitive science theories [19, 20, 21]. However, the impact of the ability of humans to interact with LLMs on the performance of human-AI collaboration remains underexplored, and the specific skills required to collaborate with LLMs efectively have yet to be identified. We believe that it is crucial to evaluate the overall performance of human-AI collaboration, considering both AI capabilities and human skills. For example, some users may be better at extracting high-quality answers from LLMs, whereas others may struggle to engage in conversations with them efectively. Thus, the performance of human-LLM collaboration likely varies depending on the communication and interpersonal skills of the user. If human skills influence collaboration with LLMs, this insight could enable applications such as skills training with LLMs or personalized adjustments based on user skills. Conversely, if human skills have

CEUR Workshop

ISSN1613-0073

Measuring the interpersonal skills

Self-assessment of six interpersonal skills ! H1 H2 ・・・

✔

High skill

Low skill

… ?

Measuring the enhancement score

A Comparison of human alone (w/o AI) and human-AI collaboration (w/ AI)

in four tasks e.g., How can 11 oranges be distributed equally among three people?

3toanedac2h/.3 +4 Make juice.

H1 H2 Distribute as 3, 4, and 4. little efect, LLMs can help to mitigate the skill disparities among users. Therefore, investigating the impact of human skills on collaboration with LLMs contributes to human-AI collaboration.

This study investigates the impact of human interpersonal skills, namely the ability to interact with other humans efectively, on the performance of human-AI collaboration. Furthermore, given that generative AI functions as an interactive conversational agent, interpersonal skills are particularly critical in determining collaborative outcomes. To explore this, we analyze the relationship between interpersonal skills and the quality of interaction in human-AI collaboration. Specifically, this study experimentally investigates whether individuals with strong interpersonal skills are more efective in collaborating with AI. Fig. 1 presents an overview of this study. We designed four types of tasks for collaboration with AI, namely knowledge, language, problem-solving, and debate tasks, and six interpersonal skills based on psychological scales. A total of 24 combinations (four tasks and six skills) were examined, and the performance of the participants and ChatGPT individually was compared with their combined performance. Our study enhances the understanding of human-AI collaboration by focusing on the role of interpersonal skills. These findings provide valuable insights for designing personalized AI systems that can adapt to diverse user skill levels.

2. Method

First, we define interpersonal skills and the enhancement score, as measured through experiments with 60 participants. We then investigate the correlations between interpersonal skills and the enhancement scores.

Interpersonal skills include social skills for building relationships [22, 23] and communication skills [24, 25]. Some conceptual overlap exists among these scales, which makes it dificult to define them as distinct interpersonal skills on their own. Therefore, this study uses the validated questionnaire-based scale ENDCOREs [26]. This scale defines communication skills in a hierarchical structure comprising three basic skills (self control, expressiveness, and decipher ability) and three interpersonal skills (assertiveness, other acceptance, and regulation of interpersonal relationships). These six skills are defined as main skills, each of which has four sub-skills. In this study, the six main skills are defined as interpersonal skills and are referred to as self-control, expressive, deciphering, assertive, other acceptance, and regulation of interpersonal relationship skills. Each interpersonal skill is evaluated using the average of the scores for the four sub skills of each main skill.

We designed four tasks based on a case involving the exchange of information and opinions, which is a typical activity in human interactions: knowledge, language, problem-solving, and debate tasks. Each task consists of two questions.

Knowledge task: This task is designed to test general knowledge. The participants were asked to arrange the four events (one of them is fictitious) in chronological order. The (∈ 1, 2, 3, 4)-th oldest event was answered by selecting from the five options: 1, 2, 3, 4, or N/A. Answers to this task were scored one point for each correct choice. Therefore, the maximum score for each question is 4 points.

Language task: This task is designed to test the comprehension of word meanings and concepts. An example question is ‘In what ways are “desks” and “chairs” similar?’ Answers are open-ended and the participants were asked to list three answers that they were confident in. We manually grouped answers that were based on the same underlying idea, and each answer group was scored for uniqueness and similarity on a 1 (low)–4 (high) scale via crowdsourcing. The scoring for this task was calculated as the sum of the scores from these two axes. Therefore, the maximum score for each question is 24 points (eight points × three answers).

Problem-solving task: This task is designed with reference to a book about lateral thinking [27] to test the skill of thinking creatively. For example, ‘List three ways to distribute 11 oranges evenly among 3 people’. Answers are open-ended and the participants were asked to list three answers that they were confident in. We manually grouped answers that were based on the same underlying idea, and each answer group was scored for uniqueness and efectiveness on a 1 (low)–4 (high) scale via crowdsourcing. The scoring for this task was calculated as the sum of the scores from these two axes. Therefore, the maximum score for each question is 24 points (eight points × three answers).

Debate task: This task is designed to test comprehension and analytical skills. An example question is ‘List opinions in favor of hosting the 2025 Osaka Expo.’ Answers are open-ended and the participants were asked to list three answers that they were confident in. We manually grouped answers that were based on the same underlying idea, and each answer group was scored for uniqueness and convincingness on a 1 (low)–4 (high) scale via crowdsourcing. The scoring for this task was calculated as the sum of the scores from these two axes. Therefore, the maximum score for each question is 24 points (eight points × three answers).

Specifically, is calculated as follows: = + − + 1 max − + 1 , where denotes the enhancement score for the task . and + denote the total scores for all questions in task when answered by humans alone and when answered in collaboration with AI, respectively. max denotes the maximum value that the score can reach in task . To prevent division by zero, 1 is added to both the numerator and the denominator. represents the extent to which the use of AI brings the score closer to the full mark, with a value closer to 1 indicating better performance.

We evaluated the correlation between each interpersonal skill and the enhancement score . The interpersonal skills consist of six skills, and the enhancement score is defined for each of the four tasks, resulting in 24 correlation coeficients (four tasks × six skills).

3. Experiments

We conducted in-person experiments to investigate the impact of interpersonal skills on interaction with AI. This study was reviewed and approved by our institutional review board (2023-I-45). 3.1. Setup and procedure A total of 60 Japanese laypeople (30 males and 30 females) in their 20s participated in the experiments from July 23, 2024, to August 9, 2024. Each participant was paid 5,000 yen as remuneration. The participants answered the questionnaire of the ENDCOREs psychometric scale with a total of eight questions (two questions for each task). In the task, participants first answered each question on their own (w/o AI) and then answered it again using ChatGPT (w/ AI). They repeated the process for the eight questions and answered a total of 16 times. To ensure uniform understanding, participants looked Scores of ChatGPT-4o-mini alone and the mean and standard deviation of and + for each task. ChatGPT alone demonstrated superior performance in the knowledge and debate tasks, whereas humans alone outperformed in the language and problem-solving tasks.

Knowledge Language Problem-solving Debate

ChatGPT 4.0 3.2. Results and discussion 3.2.1. Does ChatGPT enhance human performace? The analysis focused on the results from 58 out of 60 participants who used ChatGPT 4o-mini. No significant correlation was observed between the frequency of ChatGPT use and the enhancement scores. The correlation coeficient was also near zero, suggesting that the participants exhibited consistent enhancement scores regardless of how often they engaged with AI. The results for each task are shown in Table 1. As a result, human performance was enhanced in tasks where ChatGPT alone outperformed humans. However, in tasks where ChatGPT performed poorly on its own, collaboration led to a decline in scores. 3.2.2. Correlation between skills and enhancement scores We calculated the Pearson correlation coeficients between the six skills in ENDCOREs and the enhancement scores for each task (Table 2). The results demonstrated significant negative correlations between the knowledge task and expressive skill ( = −0.35 , < 0.01 ), knowledge task and deciphering skill ( = −0.27 , < 0.05 ), knowledge task and assertive skill ( = −0.26 , < 0.05 ) and language task and self-control skill ( = −0.27 , < 0.05 ), whereas significant positive correlations between problem-solving task and expressive skill ( = 0.31 , < 0.05 ). The results showed significant negative correlations between some skills, particularly in the knowledge task, which may indicate that people with lower interpersonal skills benefit more from AI.

k s a t e edg Et l w o n K k s a t e agu Et g n a L k s a t g n i v loS Et m e l b o r P k s a t te t a E b e D

Low High

Low High

Low High

Low High

Low High 3.2.3. Enhancement scores of Low/High skill groups The relationship between interpersonal skills and enhancement scores was examined, as illustrated in Fig. 2. The participants were categorized into two groups based on their interpersonal skill scores: “low skill” (scores of 4 or below, representing poor or neither poor nor good) and “high skill” (scores above 4, indicating high proficiency). In terms of the combinations that showed a correlation, when the results for humans alone were lower than those for ChatGPT alone, such as the knowledge task, individuals with lower interpersonal skills tended to have higher enhancement scores, with a more concentrated distribution. Conversely, when the results for humans alone were higher than those for ChatGPT alone, such as the problem-solving task, those with higher interpersonal skills tended to have higher enhancement scores. Overall, the results indicated that individuals with lower interpersonal skills benefited more from AI.

We analyzed the correlation between interpersonal skills and the number of answers that difered before and after using ChatGPT. In this analysis, the participants were divided into two groups: highconfidence, believing their performance was superior to ChatGPT, and low-confidence, believing ChatGPT was superior. A changed answer was defined as a answer that difered between not using and using ChatGPT. For the knowledge task, we counted the number of reordered events. For the other tasks, we grouped similar answers and counted the changes in the number of groups. As a result, a significant negative correlation was observed, particularly in the low-confidence group: between the problem-solving task and expressive skill ( = −0.50 , < 0.05 ), deciphering skill ( = −0.50 , < 0.05 ), other acceptance skill ( = −0.64 , < 0.01 ), and regulation of interpersonal relationship skill ( = −0.51 , < 0.05 ).

In addition, we analyzed the correlation between interpersonal skills and the number of dialogue steps as well as between interpersonal skills and the total number of words in the prompts, but no significant correlations were observed. Furthermore, there were also no significant diferences between interpersonal skills and the words used in the prompts.

These findings suggest that people with lower interpersonal skills are more likely to accept the AI’s responses when they believe that it is superior. Therefore, in the knowledge task in which ChatGPT alone performed relatively well, the score with ChatGPT improved. Conversely, in the problem-solving task, where the performance of ChatGPT was relatively lower, it is likely that the score with ChatGPT decreased because participants adopted answers generated by ChatGPT instead of their own. However, a negative correlation was also observed in the other acceptance skill, which refers to the skill of accepting others’ positions and opinions. These findings require further investigation to clarify of the underlying reasons. 3.3. Limitation The study was limited to experiments using ChatGPT-4o mini, which restricts the generalizability of the findings to other models. Also, this study specifically focused on tasks emphasizing search-like use cases, leaving creative and domain-specific tasks, such as those in medicine or artistic content creation, unexplored and limiting the understanding of human-AI collaboration in these contexts. In addition, the validity of the designed tasks, the order efects of the tasks, and other such confounding variables must be critically evaluated to ensure their appropriateness for assessing human-AI collaboration. Furthermore, the participants were limited to Japanese in their 20s. The results are based on a correlation analysis, and more detailed qualitative analysis and statistical testing are required to further our understanding.

4. Conclusion

In this study, we investigated the impact of human skills on collaboration with AI, focusing on the interpersonal skills of users. Our results showed that people with lower interpersonal skills were more likely to accept AI-generated responses, suggesting that they benefit more from AI. However, if AI’s task performance is lower than that of humans, there is a risk that collaborative performance may also decline. These findings suggest that interpersonal skills could influence how users critically assess AI-generated content. Furthermore, the extent to which AI enhances human task performance varies depending on users’ interpersonal skills, implying that there is compatibility between humans and LLMs. It is important to design AI systems to accommodate diverse user skills to achieve better human-AI collaborations.

Finally, there is much room for future studies. Although this study compared individual performance in simple with/without AI settings, real-world settings often involve more complex teaming, such as a human joining an AI team and vice versa. Considering such potential settings, this study presented the basic framework for an AI-integrated teaming evaluation.

Acknowledgments

This study was supported by collaborative research funding from LY Corporation and Cross-ministerial Strategic Innovation Promotion Program (SIP)” on “Integrated Health Care System” (grant JPJ012425).

Declaration on Generative AI

During the preparation of this work, the authors used ChatGPT, Grammarly in order to: Grammar and spelling check, Improve writing style, Text Translation, Paraphrase and reword. After using this tool/service, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.

References

[1] N. Case , How To Become A Centaur, Journal of Design and Science ( 2018 ). https://jods.mitpress.mit.edu/pub/issue3-case.

[2] E. Muller , How ai-human symbiotes may reinvent innovation and what the new centaurs will mean for cities , Technology and Investment 13 ( 2022 ) 1 - 19 .

[3] Z. Ding , A. Smith-Renner , W. Zhang , J. Tetreault , A. Jaimes , Harnessing the power of LLMs: Evaluating human-AI text co-creation through the lens of news headline generation , in: Findings of the Association for Computational Linguistics: EMNLP 2023 , 2023 , pp. 3321 - 3339 . URL: https: //aclanthology.org/ 2023 .findings-emnlp. 217 . doi: 10 .18653/v1/ 2023 .findings- emnlp.217.

[4] Z. Zhao , S. Song , B. Duah , J. Macbeth , S. Carter , M. P. Van , N. S. Bravo , M. Klenk , K. Sick , A. L. S. Filipowicz , More human than human: Llm-generated narratives outperform human-llm interleaved narratives , in: Proceedings of the 15th Conference on Creativity and Cognition , CC ' 23 , 2023 , pp. 368 - 370 . URL: https://doi.org/10.1145/3591196.3596612. doi: 10 .1145/3591196.3596612.

[5] M. Kazemitabaar , J. Chow , C. K. T. Ma , B. J. Ericson , D. Weintrop , T. Grossman, Studying the efect of ai code generators on supporting novice learners in introductory programming , in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23 , Association for Computing Machinery, New York, NY, USA, 2023 . URL: https://doi.org/10.1145/3544548.3580919. doi: 10 .1145/3544548.3580919.

[6] J. Liu , Y. Yao , P. An , Q. Wang , Peergpt: Probing the roles of llm-based peer agents as team moderators and participants in children's collaborative learning , in: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA '24 , Association for Computing Machinery, New York, NY, USA, 2024 . URL: https://doi.org/10.1145/3613905.3651008. doi: 10 .1145/3613905.3651008.

[7] I. Mondal , M. Yuan , A. N , A. Garimella , F. Ferraro , A. Blair-Stanek , B. Van Durme , J. BoydGraber , ADAPTIVE IE: Investigating the complementarity of human-AI collaboration to adaptively extract information on-the-fly , in: O. Rambow , L. Wanner , M. Apidianaki , H. Al-Khalifa , B. D. Eugenio , S. Schockaert (Eds.), Proceedings of the 31st International Conference on Computational Linguistics , Association for Computational Linguistics, Abu Dhabi, UAE , 2025 , pp. 5870 - 5889 . URL: https://aclanthology.org/ 2025 .coling-main. 392 /.

[8] X. Wang , H. Kim , S. Rahman , K. Mitra , Z. Miao , Human-llm collaborative annotation through efective verification of llm labels , in: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI '24 , Association for Computing Machinery, New York, NY, USA, 2024 . URL: https://doi.org/10.1145/3613904.3641960. doi: 10 .1145/3613904.3641960.

[9] C. Rastogi , M. Tulio Ribeiro , N. King , H. Nori , S. Amershi , Supporting human-ai collaboration in auditing llms with llms , in: Proceedings of the 2023 AAAI/ACM Conference on AI , Ethics , and Society, AIES ' 23 , 2023 , p. 913 - 926 . URL: https://doi.org/10.1145/3600211.3604712. doi: 10 .1145/ 3600211.3604712.

[10] M. Vaccaro , A. Almaatouq , T. Malone, When combinations of humans and ai are useful: A systematic review and meta-analysis , Nature Human Behaviour ( 2024 ). doi:https://doi.org/ 10.1038/s41562- 024- 02024- 1.

[11] V. Vats , M. B. Nizam , M. Liu , Z. Wang , R. Ho , M. S. Prasad , V. Titterton , S. V. Malreddy , R. Aggarwal , Y. Xu , et al., A survey on human-ai teaming with large pre-trained models , arXiv preprint arXiv:2403.04931 ( 2024 ).