Challenges in Adopting LLaMA: An Empirical Study of Discussions on Stack Overflow Ramita Deeprom1,* , Shiyu Yang2 , Yoshiki Higo2 , Morakot Choetkiertikul1 and Chaiyong Ragkhitwetsagul1 1 Faculty of Information and Communication Technology, Mahidol University, 999 Phuttamonthon 4 Road, Salaya, Nakhon Pathom 73170 THAILAND 2 Graduate School of Information Science and Technology, Osaka University 1-5, Yamadaoka, Suita, Osaka, 565-0871, Japan Abstract LLaMA (Large Language Model Meta AI) has quickly gained traction among developers due to its wide-ranging applications and its capabilities to be integrated into software projects. As interest in LLaMA grows, discussions around it have surged on platforms like Stack Overflow. The developer community, with its collaborative nature, serves as a valuable source for studying LLaMA’s quality, its emerging trends, and insights into its usage. Despite this growing attention, there has been no comprehensive study examining how the community interacts with and discusses LLaMA. This study addresses that gap by exploring conversations on Stack Overflow related to LLaMA and its quality, with the objective of identifying key themes and recurring patterns in these discussions. We systematically collected and analyzed 473 posts from Stack Overflow that contained the keyword “LLaMA” or were tagged accordingly. The analysis revealed that prominent topics of discussion include model configuration, error handling, and integration with other technologies. Furthermore, we identified frequent co-occurring tags, underscoring LLaMA’s integration within the larger ecosystem of large language models and its interoperability with widely used frameworks, such as Python and Hugging Face Transformers. The findings highlight the complexity of working with LLaMA, especially in model configuration and fine-tuning, indicating a need for better resources, documentation, and community support. The study also suggests that future development should prioritize interoperability with popular machine-learning frameworks to improve the LLM’s quality and to strengthen LLaMA’s role in the AI ecosystem. Keywords LLaMA, Stack Overflow, Large Language Models’ Quality 1. Introduction platforms like Stack Overflow (SO),2 an online community where developers ask questions, share knowledge, and pro- The rapid advancements in artificial intelligence (AI) have vide solutions related to software development and technol- revolutionized the field of technology, leading to the cre- ogy. SO has become one of the most widely used platforms ation of powerful large language models (LLMs) that are for developers to collaborate, troubleshoot, and learn from transforming how developers and organizations approach each other, making it a rich source of information about problem-solving. One such model is Meta’s LLaMA1 , an real-world challenges and practical applications of various open-source LLM that has garnered substantial attention technologies. Studying SO is essential because it reflects the from the developer community [1]. Unlike many propri- collective experiences and expertise of a global community etary models, LLaMA offers developers the flexibility to of developers, providing valuable insights into the quality, fine-tune and customize the model for specific use cases, common issues, and trends that arise with new technologies making it an attractive alternative for those who require like LLaMA. By examining the discussions on SO, we can more control and adaptability in their applications [1]. better understand not only the key themes and challenges Recent studies have demonstrated LLaMA’s superior per- developers face with LLaMA but also the broader context formance in specific domain tasks, such as cheminformat- of its integration and adoption in various fields. This under- ics, where it has outperformed models like ChatGPT in standing is critical for identifying areas where additional tasks such as SMILES embeddings for predicting molecu- support, documentation, or tools might be needed to im- lar properties and drug-drug interactions (DDI) [2]. This prove the developer experience and further promote the suggests that LLaMA is particularly effective in tasks that effective use of LLaMA. demand high degree of precision and domain-specific ex- This study aims to address an initial gap by conducting pertise, setting it apart from other LLMs. While models an empirical analysis of Stack Overflow posts tagged with like ChatGPT, Bard, and Ernie may offer unique features LLaMA to identify the predominant discussion topics related such as real-time web access or higher computational effi- to its quality and adoption, and associated technologies. By ciency, LLaMA stands out by providing a well-rounded bal- employing keyword frequency analysis and categorizing the ance across various criteria, making it suitable for a broader posts, this study seeks to answer two key research questions: range of applications [3]. (1) What are the main topics of discussion regarding LLaMA The growing interest in LLaMA is particularly evident on on Stack Overflow? and (2) What related themes emerge in QuASoQ 2024: 12th International Workshop on Quantitative Approaches these discussions? Through this initial analysis, we aim to to Software Quality, December 03, 2024, Chongqing, China provide early insights into the specific challenges developers * Corresponding author. face, the solutions they seek, and the broader implications $ ramita.dep@student.mahidol.ac.th (R. Deeprom); for LLaMA’s role within the AI ecosystem. The findings yangsy@ist.osaka-u.ac.jp (S. Yang); higo@ist.osaka-u.ac.jp (Y. Higo); from this research study will serve as a foundation for a morakot.cho@mahidol.ac.th (M. Choetkiertikul); chaiyong.rag@mahidol.ac.th (C. Ragkhitwetsagul) more comprehensive future study, contributing valuable € https://ysy-dlg.github.io/MyHomePage (S. Yang); insights to both practitioners and researchers as we further https://sites.google.com/view/yhigo/home (Y. Higo); our understanding of LLaMA’s use and integration within https://morakotch.wordpress.com/ (M. Choetkiertikul); diverse technical environments. https://cragkhit.github.io/ (C. Ragkhitwetsagul) The structure of this paper is as follows. Section 2 pro- © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 2 https://github.com/meta-llama/llama https://stackoverflow.com/ CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings 35 vides the background and related work, detailing prior re- LLMs. Similarly, comparative analyses have shown that search on the adoption of large language models (LLMs) while ChatGPT and other models like Bard and Ernie offer such as LLaMA and their application in real-world scenar- advantages in certain areas, such as real-time internet ac- ios. The methodology employed in our research, including cess or computational efficiency, LLaMA provides balanced data collection and preprocessing techniques, is explained performance across multiple criteria, making it a versatile in Section 3. Section 4 presents the results of our empirical tool for various applications [3]. study, focusing on the analysis of Stack Overflow discus- Moreover, the performance of Llama 2 has been noted sions to answer the research questions posed in this study. to exhibit minimal variation across different languages, of- We then discuss the implications of our findings in Section 5, fering consistency in sentiment analysis tasks. However, where we highlight the key challenges faced by developers this consistency sometimes comes at the cost of skewing when working with LLaMA and suggest potential improve- ratings towards positive sentiment, even in scenarios where ments for future development. Finally, Section 6 concludes more nuanced interpretations are required [9]. Further- the paper and outlines potential avenues for future research, more, recent studies on job recommendations generated such as expanding the dataset and exploring more advanced by LLaMA reveal both strengths and limitations. While stages of LLaMA adoption. LLaMA suggests a wider variety of professions compared to ChatGPT, its recommendations often include impractical or nonsensical roles, reflecting a trade-off between diversity 2. Background and Related Work and practicality [10]. This indicates the need for improved prompt engineering and bias mitigation in LLM applications The rapid adoption of generative AI, particularly large lan- to ensure fairer and more relevant outcomes across diverse guage models (LLMs), has sparked significant interest in user groups. understanding how users are integrating these tools into Several studies have leveraged Stack Overflow data to their workflows. Previous research shows that many profes- analyze trends within the developer community, providing sionals increasingly rely on generative AI, such as ChatGPT insights into quality, common challenges, emerging tech- and LLaMA, to solve problems traditionally addressed on nologies, and evolving developer needs. Silva et al. [11] platforms like Stack Overflow (SO) [4, 5]. This shift sug- report that ChatGPT has significantly impacted SO, offering gests a change in the problem-solving paradigm, where fast, human-like responses that have raised questions about AI-generated solutions are becoming a first resort for many the platform’s future in the AI era. The study noted a decline developers, streamlining the troubleshooting process and in overall SO activity, though some communities remain ac- improving efficiency [4]. However, despite the growing re- tive. Both models excel at addressing general programming liance on AI, recent studies indicate that not all users are queries but struggle with specific frameworks and libraries, fully satisfied with AI-generated responses. Some develop- leading developers to return to SO when LLMs fall short. ers still face challenges, particularly with complex technical Similarly, Zhong et al. [6] developed the RobustAPI dataset, issues, prompting them to seek human-based community featuring 1,208 coding questions from SO related to 18 Java support on platforms like SO [6, 5]. This highlights the limi- APIs. Their study revealed that even advanced models like tations of AI models in delivering contextually accurate and GPT-4 produced API misuses in 62% of the generated code, reliable answers for more nuanced problems [7, 5]. posing risks when applied to real-world software develop- LLaMA, an open-source LLM created by Meta, offers no- ment. table advantages that contribute to its rising popularity Nonetheless, there is no study that investigates the quality within the developer community. Released to the public of LLaMA and its adoption in practice. This study fills in in February 2023, with LLaMA 3.1 debuting in July 2024, the gap by studying the discussions related to LLaMA on the model has garnered over 300 million downloads glob- SO discussions. ally, underscoring its widespread adoption [8]. Compared to ChatGPT, LLaMA is perceived as more complex to install and configure, yet its appeal lies in its ability to provide 3. Methodology fine-tuned, context-specific outputs, making it particularly attractive to developers who require precision and control As shown in Figure 1, a motivating example is a Stack [8, 3]. Furthermore, LLaMA’s enhanced security features Overflow post where a user inquires about installing the and the ability to be hosted internally within organizations LLaMA-cpp-python package. This post has garnered without the risk of leaking sensitive information make it a 38,975 views (at the time of writing), illustrating the strong contender for enterprise use cases [3]. These charac- widespread interest in LLaMA but also highlighting that teristics reduce the risk of biased outputs, which is often a developers frequently encounter challenges requiring exter- concern for beginners relying too heavily on AI-generated nal help. Despite its growing popularity, the installation and responses [3]. The model’s open-source nature also allows configuration of LLaMA packages remain common stum- for greater flexibility in integration and customization, of- bling blocks. fering experienced developers a robust tool for specialized In light of this, our research focuses on examining the applications [3, 2]. discussions surrounding LLaMA on Stack Overflow. By Studies have highlighted that LLaMA excels in certain analyzing these interactions, we aim to uncover the most domain-specific tasks, such as cheminformatics, where it prevalent issues and limitations faced by developers work- outperforms ChatGPT in Simplified Molecular Input Line ing with LLaMA. This study not only seeks to identify key Entry System (SMILES) embeddings for molecular property challenges but also offers valuable insights for both novice and drug-drug interaction (DDI) predictions [2]. This su- users looking to get started with LLaMA and experienced perior performance suggests that LLaMA is well-suited for developers seeking to optimize and enhance their imple- tasks that require high degree of precision and the handling mentations. Ultimately, our findings will contribute to im- of specific domain data, further distinguishing it from other proving the support and resources available to the LLaMA 36 Figure 1: A LLaMA question of Stack Overflow (Post ID 77267346) community, facilitating smoother adoption and integration As a result, we adopted a more direct and up-to-date data of the model into various workflows. collection approach. We utilized the web scraping tool4 to We ask the following research questions in this study. scrape posts directly from Stack Overflow. The scraping process was conducted on July 22, 2024. To comply with 1. RQ1: What are the topics of discussion about LLaMA Stack Overflow’s usage policies and avoid overloading their on Stack Overflow? We desire to identify and cate- servers, we incorporated waiting times between requests. gorize the topics of discussion related to “LLaMA” The data collected included the posts’ links, titles, bodies, on Stack Overflow. This is to determine the most and tags. common themes and issues raised by the developer To effectively capture posts related to “LLaMA”, we em- community concerning LLaMA. ployed two distinct methods: 2. RQ2: What are the related topics when discussing Method 1: Keyword Search — We conducted a search LLaMA on Stack Overflow? The second research on Stack Overflow using the keyword “LLaMA”5 . This search question focused on identifying related tags co- yielded 2,405 posts, which we categorized as follows: occurring with the LLaMA tag on Stack Overflow. This is to find other relevant topics or challenges • Title Group (644 posts): Posts where “LLaMA” ap- that LLaMA users may face or need to study. peared in the title. • Body Group (1,761 posts): Posts where “LLaMA” This section details the steps undertaken to address the appeared in the body. However, after manual inspec- two research questions posed earlier. As illustrated in Figure tion, many of these posts were deemed irrelevant 2, our methodology involves three key phases: data collec- and thus excluded from further analysis. tion, preprocessing, and analysis. Each phase is designed to ensure a systematic and thorough examination of Stack Method 2: Tag Search (770 posts) — We also searched Overflow discussions related to LLaMA. In the data collec- for posts tagged with “LLaMA” on Stack Overflow6 . This tion phase, we gathered relevant posts from Stack Overflow, search resulted in 770 posts, which were compiled into a ensuring a representative sample of developer interactions. separate group called the Tag Group. This was followed by the preprocessing phase, where we cleansed and refined the data to ensure its quality and rel- evance for analysis. Finally, the analysis phase involved 3.2. Data Preprocessing categorizing the posts and performing keyword frequency Data preprocessing was essential to ensure the relevance analysis to uncover common themes and patterns. and quality of the data used in our analysis. The following steps were undertaken to refine the data: 3.1. Data Collection Step 1: Tag Separation — The tags in the Tag Group were initially compiled as a single string. To analyze the tags Our study is based on data collected directly from Stack associated with each post more precisely, we separated them Overflow, particularly focusing on posts related to LLaMA, into individual tags, enabling more effective identification the generative AI model from Meta. Initially, we considered and analysis. using the Stack Overflow public data dump files, including Step 2: Duplicate Removal — During preprocessing, we Posts.xml and Tags.xml3 . However, after downloading identified overlaps between the Title Group and Tag Group, and inspecting these files, we found that they did not con- as some posts appeared in both groups due to being tagged tain recent posts relevant to our study, particularly those involving technologies like LLaMA, likely due to the release 4 Web Scraper version 1.87.6 (available at: https://webscraper.io/) of LLaMA being more recent than the last update of the data 5 We queried from the URL https://stackoverflow.com/search?tab= dump. newest&q=LLaMA&searchOn=3 6 We queried from the URL https://stackoverflow.com/questions/tagged/ 3 https://archive.org/details/stackexchange LLaMA?tab=Newest 37 Figure 2: The Experimental Procedure with “LLaMA.” Additionally, we detected duplicate entries Handling and Debugging, Installation and Setup Issues, Inte- with identical post links and titles. These redundancies gration and API Usage, Runtime and Performance Issues, and were removed, resulting in a refined dataset of 473 posts Model Deployment and Hosting. These six groups were estab- comprising 395 posts tagged with “LLaMA” and 78 posts lished before the manual classification by the first authors without the tag. during the data collection and data preprocessing steps. One post could fall into multiple categories. Any disagreements 3.3. Dataset Characteristics were resolved through discussion until a consensus was reached. After data collection and preprocessing, our final dataset RQ2: What related topics and technologies are as- consisted of 473 posts, all centered on LLaMA-related topics. sociated with LLaMA? — We examined the tags associ- These posts cover a range of issues, questions, and discus- ated with “LLaMA” to identify related topics and technolo- sions about LLaMA, including configuration, usage, and gies. The co-occurrence of these tags with “LLaMA” shows challenges. the broader technological ecosystem and application areas For instance, a typical post in our dataset may include a linked to LLaMA. query about fine-tuning the LLaMA model: “How do I fine-tune the LLaMA model on a 4. Results custom dataset? I’m facing memory issues during training and could use some advice on This section presents the findings from our analysis of the optimizing performance.” discussions related to the LLaMA model on Stack Overflow and the answers to our research questions. We address Another example might address integration issues: the research questions (RQ1 and RQ2) through a detailed “I’m trying to integrate LLaMA with an exist- examination of the collected and cleansed datasets. ing API but keep encountering errors during the authentication process. Has anyone faced 4.1. Answering RQ1 similar issues?” To answer RQ1, we manually categorized the posts into These examples illustrate the types of discussions that six distinct categories based on the nature of the issues dis- form the basis of our subsequent analysis. cussed. To assess the reliability of the manual classification, we calculated the inter-rater reliability using Cohen’s Kappa statistic. The Kappa score was 0.883, indicating an almost 3.4. Data Analysis perfect agreement between the two authors. The catego- Using the cleansed datasets, we analyzed the topics of dis- rization helped us to identify the most common themes in cussion related to LLaMA on Stack Overflow to address our the developer community’s conversations about LLaMA. Ta- research questions: ble 1 provides a summary of the categories and the number RQ1: What are the common topics discussed regard- of posts that relate to each category. ing LLaMA? — We manually classified the titles and bodies From our analysis, it is evident that the majority of discus- of the posts to identify common topics. To ensure thorough- sions focus on Model Configuration and Fine-Tuning, with ness, the first author initially skimmed through all posts to 135 posts, making it the most frequently discussed topic. get a sense of the themes and formulated the six categories This suggests that many developers are struggling with con- as a preliminary structure. Then the first and second au- figuring and fine-tuning LLaMA models to meet specific thors independently reviewed all posts, categorizing them needs. Posts in this category often mention challenges such into six groups: Model Configuration and Fine-Tuning, Error as adjusting hyperparameters, loading pre-trained models, 38 Figure 3: Example of Model Configuration and Fine-Tuning post (Post ID 76880690) Table 1 challenges, and difficulties in setting up dependencies. The Categories of LLaMA Discussion on Stack Overflow high number of posts in this category indicates that getting started with LLaMA can be particularly challenging, espe- Category Number of Posts cially for users who are new to the model or unfamiliar with Model Configuration and Fine-Tuning 135 the broader ecosystem of tools it integrates with. Figure Error Handling and Debugging 110 5 shows a Stack Overflow post titled “Cuda 12.2 and issue Installation and Setup Issues 91 with bitsandbytes package installation” categorized under Integration and API Usage 86 “Installation and Setup Issues.” In this post, the developer is Runtime and Performance Issues 73 facing an issue with running Llama 2 on Google Colab and Model Deployment and Hosting 24 asks for help. Total 519 Integration and API Usage, with 86 posts, reflects discus- sions on how to connect LLaMA with other systems, par- ticularly through APIs. Developers often seek guidance and optimizing models for specific tasks or datasets. The on integrating LLaMA into existing workflows, leveraging prevalence of this category suggests that LLaMA’s flexibility its capabilities alongside other tools, and addressing API- and complexity in configuration require careful attention related challenges. These discussions highlight the impor- and often lead to challenges that developers seek to over- tance of seamless integration between LLaMA and other come. Posts in this category commonly address issues like technologies, as well as the need for clear guidelines on API adjusting hyperparameters, loading pre-trained models, and usage. optimizing models for particular tasks or datasets. The Runtime and Performance Issues, comprising 73 posts, fo- prominence of this category indicates that LLaMA’s flexibil- cuses on challenges that developers face during the execu- ity and complexity in configuration often present challenges tion of LLaMA models. This includes discussions on optimiz- that developers actively seek to resolve. Figure 3 shows a ing model performance, managing resource consumption, Stack Overflow post titled “Chat with spreadsheet using and addressing latency issues. Posts in this category often Meta Llama (Llama 2 13B Chat HF),” categorized under the highlight the need for efficient execution of LLaMA models, Model Configuration and Fine-Tuning category. In this post, especially in production environments where performance the questioner is facing the problem of using LLaMA for is critical. querying spreadsheet data. Model Deployment and Hosting, with 24 posts, is the least Error Handling and Debugging, accounting for 110 posts. discussed category. Posts here focus on deploying LLaMA This category includes posts where developers encountered models into production, managing model versions, and host- errors during the use of LLaMA and sought solutions to re- ing models on different platforms. The relatively low num- solve these issues. Common topics in this category involve ber of posts in this category might suggest that deployment troubleshooting runtime errors, resolving compatibility is- is a more advanced stage of working with LLaMA, which sues with other libraries, and debugging scripts that fail to fewer users have reached, or that deployment-related issues execute as expected. The prevalence of posts in this cate- are less frequent or already well-documented within the gory underscores the need for robust debugging tools and community. clear documentation to help developers efficiently resolve Overall, the distribution of posts across these categories issues. Figure 4 depicts a Stack Overflow post titled “How provides valuable insights into the areas where LLaMA users to debug the Llama 2 inference command with VSCode,” are most likely to encounter challenges. It also highlights which is categorized under “Error Handling and Debugging.” the importance of comprehensive support and resources In this post, the questioner asks about configuring Visual in the areas of model configuration, error handling, and Studio Code to debug the Llama 2 inference script. integration. Installation and Setup Issues is another prominent cate- gory, comprising 91 posts. This category covers problems en- countered during the initial stages of working with LLaMA, including installation errors, environment configuration 39 Figure 4: Example of the Error Handling and Debugging post (Post ID 77421713) Figure 5: Example of Installation and Setup Issues post (Post ID 78194505) 4.2. Answering RQ2 learning tools. LangChain, in particular, is a framework designed for building applications with LLMs, suggesting To address RQ2, we examined the co-occurrence of tags that LLaMA users are developing complex workflows that in posts discussing LLaMA. By analyzing these tags, we involve multiple LLMs. aimed to identify related topics and technologies that are Notably, the openai-api tag appeared in 26 posts, indicat- commonly mentioned alongside LLaMA on Stack Overflow. ing a significant interest in interoperability between LLaMA Table 2 summarizes the frequency of the most common and OpenAI’s models. The posts in this category reveal co-occurring tags. several common themes: The analysis revealed that the large-language-model tag was the most frequently co-occurring tag with LLaMA, ap- 1. Interoperability Between LLaMA and OpenAI Mod- pearing in 201 posts. This suggests that discussions around els: Many posts discuss how to integrate or migrate LLaMA are often framed within the broader context of large between LLaMA models and OpenAI APIs. For in- language models, indicating that developers are considering stance, questions related to migrating from ChatGPT LLaMA alongside other major models in this category. The to Llama 2 or using different LlamaIndex chat en- frequent mention of python (184 posts) and huggingface- gine modes with an OpenAI key suggest that users transformers (109 posts) indicates that developers are ac- are exploring how to use both systems together or tively using Python-based tools and libraries, particularly comparing their functionalities. Hugging Face’s Transformers library, to work with LLaMA. 2. LangChain and LLaMA: Several posts mention This reflects LLaMA’s integration into the Python ecosys- LangChain in conjunction with LLaMA. LangChain tem and its compatibility with popular machine-learning is a framework for building applications with LLMs, frameworks. and the discussions around using it with LLaMA sug- The co-occurrence of tags like langchain (77 posts) and py- gest that users are working on sophisticated work- torch (70 posts) further supports the observation that LLaMA flows involving multiple language models. This high- is frequently used in conjunction with other machine- lights LLaMA’s role in the broader landscape of lan- 40 Table 2 LLaMA AI. This assumption may have resulted in the in- Top Co-Occurring Tags with llama Tag on Stack Overflow clusion of irrelevant or off-topic content. We mitigated this risk by performing a manual verification of 500 posts to en- Tag Occurrences sure relevance, though some less obvious irrelevant content llama 398 might still remain. Additionally, our reliance on manual clas- large-language-model 201 sification introduces the risk of human error and bias. To python 184 address this, two authors independently classified the posts, huggingface-transformers 109 and any discrepancies were resolved through discussion langchain 77 to increase consistency and reduce subjectivity. However, pytorch 70 huggingface 66 biases inherent in manual processes may still exist, and the llama-index 40 absence of automated classification tools may have limited nlp 39 the scalability of the analysis. artificial-intelligence 34 Furthermore, the data collection was conducted only up fine-tuning 28 until July 22, 2024, which excludes newer posts. As the field openai-api 26 of large language models (LLMs) evolves rapidly, this limi- machine-learning 25 tation may have prevented us from capturing recent trends ollama 23 or emerging challenges, potentially affecting the complete- llamacpp 22 ness and timeliness of our analysis. External Validity: The llama-cpp-python 22 findings are based solely on Stack Overflow (SO) posts with llama3 20 python-3.x 20 the keyword “LLaMA” in the titles or tags, which may limit amazon-sagemaker 19 the generalizability of our results to other technical Q&A chatbot 14 platforms such as GitHub, Reddit, or specialized forums gpu 14 where different types of discussions and more complex tech- amazon-web-services 12 nical issues may be addressed. By focusing exclusively on Others 356 SO, we may have missed richer, more nuanced developer Total 1,819 challenges that could provide a broader understanding of LLaMA adoption across different communities. guage model applications. 5. Implications 3. Fine-Tuning and Model Performance: With the fine- tuning tag appearing in 28 posts, this category re- The findings from this study provide valuable insights into flects discussions around optimizing LLaMA models. the quality of Meta’s LLaMA model and how the developer Posts such as “LLaMA Index training my own model community engages with it on Stack Overflow, particularly gives poor results” indicate challenges in fine-tuning in terms of overcoming technical challenges. The analysis LLaMA models. Users seek advice on improving reveals that discussions predominantly focus on issues such model performance, particularly in fine-tuning and as configuring, fine-tuning, and integrating LLaMA into optimizing models for specific tasks. various applications. This highlights the model’s flexibility 4. Vector Databases and RetrievalQA: Discussions in but also points to its complexity, underscoring the need for this area involve using LLaMA models with vector improved documentation, resources, and tools. databases and RetrievalQA. Users are focusing on One key implication is the necessity for enhanced com- effectively retrieving documents or managing stor- munity support and resources for model configuration and age when integrating LLaMA with OpenAI’s API, fine-tuning. The frequency of posts on these topics suggests reflecting the complexity of tasks users are under- that many developers, especially those without advanced taking. expertise in machine learning, encounter significant diffi- 5. Computational Resources: Questions related to hard- culties. By improving documentation and offering more ware usage, such as issues with running LLaMA user-friendly tools, Meta could lower the barrier to entry models on CPUs or optimizing GPU usage, high- for a wider audience, leading to broader adoption of LLaMA. light developers’ concerns about the computational This could also include the development of community- demands of LLaMA models. Tags such as gpu and driven forums, FAQs, or official support channels dedicated amazon-sagemaker appear alongside discussions fo- to troubleshooting configuration and fine-tuning issues. cused on resource optimization. Another important implication is the need to prioritize seamless integration with existing machine-learning ecosys- These findings illustrate that LLaMA is part of a larger tems. The co-occurrence analysis shows that LLaMA is fre- ecosystem of tools and technologies, with significant inter- quently used in conjunction with popular frameworks like est in how it can be integrated with or compared to other Hugging Face’s Transformers, PyTorch, and LangChain, par- models, particularly those from OpenAI. The discussions ticularly in Python environments. This suggests that future also underscore the importance of effective model manage- iterations of LLaMA should focus on making integration ment, performance optimization, and resource utilization with these frameworks more straightforward and efficient, when working with LLaMA. potentially through more robust APIs, pre-built connectors, or better interoperability guidelines. Ensuring compatibility 4.3. Threats to Validity with widely-used tools will be crucial in positioning LLaMA as a go-to solution for developers working on real-world Several threats to validity may impact the findings of this applications. Finally, the relatively low number of posts study. Internal Validity: One potential threat is the assump- discussing the deployment and hosting of LLaMA models tion that all posts in the dataset were relevant to Meta’s 41 suggests that this is still an emerging area. However, as 7. ACKNOWLEDGEMENT more developers move toward deploying LLaMA models in production environments, there will likely be an increasing This research project was supported by the Faculty of ICT, demand for comprehensive deployment tools, best practices, Mahidol University. and infrastructure support. References 6. Conclusion and Future Work [1] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. In conclusion, this preliminary study provides a detailed Lacha ux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, investigation of the quality, the challenges, and related top- F. Azhar, et al., Llama: Open and efficient foundation ics in the Stack Overflow community’s discussions about language models, arXiv preprint arXiv:2302.13971 LLaMA. By understanding these areas, Meta and the broader (2023). developer community can better support the use of LLaMA, [2] S. Sadeghi, A. Bui, A. Forooghi, J. Lu, A. Ngom, Can ultimately driving innovation in LLM development. large language models understand molecules?, This study provides valuable insights into the challenges 2024. URL: https://arxiv.org/abs/2402.00024. developers face when adopting LLaMA, based on Stack Over- arXiv:2402.00024. flow discussions. However, several areas for future research [3] K. Wangsa, S. Karim, E. Gide, M. Elkhodr, A systematic could significantly enrich the findings and address the limi- review and comprehensive analysis of pioneering ai tations identified in this study. First, expanding the dataset chatbot models from education to healthcare: Chat- to include posts beyond July 2024 will help capture evolving gpt, bard, llama, ernie and grok, Future Internet 16 trends as LLaMA and other large language models (LLMs) (2024). URL: https://www.mdpi.com/1999-5903/16/7/ continue to develop. Additionally, incorporating data from 219. doi:10.3390/fi16070219. other platforms such as GitHub Issues, Reddit, and developer [4] J. Son, B. Kim, Trend Analysis of Large Language forums could provide a broader perspective on LLaMA’s us- Models through a Developer Community: A Focus on age, especially on more complex technical problems and Stack Overflow, Information 14 (2023). nuanced discussions that may not be captured on Stack [5] A. Hörnemalm, O. Norberg, T. Mejtoft, ChatGPT as a Overflow alone. Comparing LLaMA to other LLMs, such as Software Development Tool The Future of Develop- ChatGPT or Claude, would also provide valuable insights, ment, Master’s thesis, Umeå University, Department allowing researchers to understand LLaMA’s challenges in of Applied Physics and Electronics, 2023. the broader landscape and better justify its focus. [6] L. Zhong, Z. Wang, Can llm replace stack overflow? a Furthermore, future research should enhance the method- study on robustness and reliability of large language ology by employing a more rigorous approach to data fil- model code generation, in: Proceedings of the AAAI tering and analysis. Pre-processing the data to exclude triv- Conference on Artificial Intelligence, volume 38, 2024, ial questions and focusing on more substantial challenges pp. 21841–21849. would yield more meaningful insights. Using established [7] K. Jin, C.-Y. Wang, H. V. Pham, H. Hemmati, Can qualitative coding frameworks for topic classification would ChatGPT Support Developers? An Empirical Evalua- further improve the transparency and validity of the analy- tion of Large Language Models for Code Generation, sis. Another promising direction is incorporating sentiment in: Proceedings of the 21st International Conference analysis to understand community attitudes toward LLaMA. on Mining Software Repositories, MSR ’24, 2024, p. By analyzing the tone of discussions across platforms, re- 167–171. searchers could uncover whether developers’ experiences [8] A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sen- with LLaMA are generally positive, negative, or neutral, of- gupta, S. Yoo, J. M. Zhang, Large Language Models for fering Meta and the developer community actionable feed- Software Engineering: Survey and Open Problems, in: back for improving the tool. ICSE-FoSE’23, 2023, pp. 31–53. Additionally, complementing the analysis with user stud- [9] A. Buscemi, D. Proverbio, Chatgpt vs gemini vs llama ies—such as surveys or interviews—could provide a deeper on multilingual sentiment analysis, 2024. URL: https: understanding of the practical challenges faced by develop- //arxiv.org/abs/2402.01715. arXiv:2402.01715. ers using LLaMA in real-world scenarios. Exploring specific [10] A. Salinas, P. Shah, Y. Huang, R. McCormack, use cases where LLaMA is integrated into different applica- F. Morstatter, The unequal opportunities of large tion domains, such as natural language processing (NLP) or language models: Examining demographic biases enterprise applications, could reveal unique challenges and in job recommendations by chatgpt and llama, in: benefits in various contexts. Finally, investigating advanced Proceedings of the 3rd ACM Conference on Eq- stages of LLaMA adoption, particularly in production envi- uity and Access in Algorithms, Mechanisms, and ronments, would help identify issues related to deployment Optimization, EAAMO ’23, Association for Com- and model hosting, offering a more complete picture of puting Machinery, New York, NY, USA, 2023. LLaMA’s practical applications and limitations. By address- URL: https://doi.org/10.1145/3617694.3623257. doi:10. ing these areas, future research will contribute to a more 1145/3617694.3623257. comprehensive understanding of LLaMA’s role within the [11] L. Da Silva, J. Samhi, F. Khomh, Chatgpt vs llama: LLM ecosystem, driving more effective support for develop- Impact, reliability, and challenges in stack overflow ers and fostering broader adoption of open-source LLMs. discussions, arXiv preprint arXiv:2402.08801 (2024). 42