Software startup is a unique type of company with unique characteristics. On the one hand, they must ofer innovative products appealing to customers to generate revenue and survive, but on the other hand, they are limited in resources, time, and experience. During the new product development, it is important to experiment with their original ideas. However, doing a meaningful experiment requires resources and challenges. A study on failed software startups shows that, despite its importance, many software startups skipped or did not experiment with their ideas. The study identifies 25 inhibitors spread in five experimentation stages. In the last few years, Large Language Models (LLMs) have become a popular technology. The advancement of LLM has made it adopted into many parts of the software development cycle. Studies show that LLM also has been used to generate new innovative product ideas and to manage innovation. However, there is no investigation into the possibility of utilizing the power of LLM to help software startups do experimentation. Interactions to an LLM are done through prompts. During the interaction or session, a user will send one or more prompts in a zero-, one-, or few-shots to an LLM agent. Unfortunately, learning and using prompts efectively requires time and resources, things that software startups are scarce with. In this project, we aim to help improve the experimentation process and address the inhibitors by leveraging the power of LLMs. There are five initial research questions and studies planned in the project. In the first step, we will investigate current experimentation practices, challenges, inhibitors, and the strategies used to circumvent them. Secondly, we will investigate how AI has been used in today's experimentation. Then, we will investigate the set of measurements available to measure the success of an experiment. The next step is to investigate how to support experimentation using LLMs followed by a validation sequence. The first form of support is a prompt guidebook to help software startups use an LLM agent to help their experimentation. The second form is an LLM-based assistant tailored specifically to guide the experimentation process.
Software startups are emerging companies in the software domain that aim to present
innovative software solutions [
Studies show that experimentation is one fundamental approach to finding a successful
and sustainable business idea [
Unfortunately, the adoption of experimentation in software startups is low. A study by
Melegati et al. [
Recently, an AI-assisted approach to developing new product ideas has emerged. The
combination of humans and AI generates new innovative ideas. Large Language Models (LLMs), a
subset of AI technology, have revolutionized natural language processing tasks by eficiently
learning linguistic patterns from vast text data. Popular LLMs, such as OpenAI’s GPTs1, Google’s
BERT2, Gemini3, and Meta’s Llama4 employ transformer architectures to generate or understand
human-like text across various applications, from chatbots to content generation. These models
have remarkable capabilities in capturing context, nuance, and semantics. Their pre-training
and fine-tuning methodology makes them versatile across diverse tasks without task-specific
model modifications [
In this project, we aim to improve the experimentation process and address the inhibitors by
1https://openai.com/research
2https://blog.research.google/2018/11/open-sourcing-bert-state-of-art-pre.html
3https://deepmind.google/technologies/gemini/introduction
4https://research.facebook.com/publications/llama-open-and-eficient-foundation-language-models/
leveraging the power of LLMs. There are five initial research questions and studies. We plan to
run the project using the design science approach [
Experimentation in software startups is critical to validate ideas and create a sustainable
business. A study based on the postmortem artifacts shows that many software startups failed
to experiment with their ideas, which, to some extent, might lead them to a disastrous state [
However, the study does not elaborate on how or what strategies the software startups tried to overcome the inhibitors. This knowledge is important as a lesson learned for future cases. Unfortunately, extending the study could be challenging since the main sources are postmortem artifacts. A similar study on the surviving software startups could fill the gap. The result would give valuable knowledge on how to address the inhibitors. This draws the first research question (RQ1), "What are the practices, challenges, inhibitors, and solutions in experimentation in software startups?"
In addition, further exploration of adopting AI technologies in experimentation could also be performed to learn how AI technology has been used and how useful they are. This knowledge will allow further investigation into what areas or aspects can be improved. This raises the second research question (RQ2), "How do software startups incorporate AI technologies, especially LLMs, in new product development?"
It is also important to have a clear guideline, framework, or set of measurements to quantify whether an experiment produces a meaningful result regardless of the consequences, continuation, or abandonment of the idea. This leads to the third research question (RQ3), "How to measure whether an experiment produces a meaningful result or not?"
LLMs have gained massive attention in the last few years due to their advancement and
potential. The vast amount of training datasets makes LLMs very powerful and knowledgeable
tools. LLMs have been integrated into many aspects of software engineering, from requirement
elicitation to testing and debugging [
The advancement of LLMs has been used in many stages in the software development cycle
[
Interactions to an LLM are done through prompts. During the interaction or session, a user
will send one or more prompts in a zero-, one-, or few-shots to an LLM agent [
The zero-shot is a prompt without any prior context or example. This means the LLM agent will try to understand the context only based on the sole prompt. In most cases, the sole prompt lacks clear context. Hence, it is very likely the zero-shot prompt will return a generic output that is less desirable for specific needs or cases.
The one-shot prompt is when the user communicates exactly one example or a context before sending the intended prompt. Compared to the zero-shot, this prompt will produce outputs closer to the expectation, a more desirable.
The few-shots prompt is when the user has supplied multiple examples or contexts before the intended prompt. This will very likely produce a more relevant output compared to the one-shot. This prompt is also called many-shots when the number of examples or context is very large.
An efective prompt strategy helps the LLMs produce the most relevant output [ 19]. However, writing prompts to leverage an LLM agent to guide the experiment process requires time and resources, things that software startups do not have. This raises another research question (RQ4), "How to efectively use prompts to guide the experimentation process?"
Moreover, the availability of previous valuable experiences from failed and survived software startups may improve the LLMs’ responses. However, it has to be admitted that LLMs can only give directions or suggestions and not direct solutions based on a probabilistic approach. The final decision lies in the hands of the software startup’s leaders. These experiences can make a better experimentation context when supplied into LLMs. In addition to that, specific information related to a project could even make a more personalized environment for betterguided experimentation. These lead to our latest research question (RQ5), "How to improve the experimentation process by leveraging the power of LLMs?"
To answer RQ1 and RQ2, we plan to conduct two parallel studies. The first one (S1) is an interview study with software startups to understand the current practices, challenges, inhibitors, and solutions for doing experiments. Furthermore, we would like to know how they manage the efect of inhibitors. The result will then be analyzed to see patterns or strategies that can be useful in future cases. We will also investigate how these software startups incorporate AI technologies in their experimentation, how satisfied they are, and what could be improved. A literature study (S2) will be executed to enrich the knowledge from previous research. We aim to publish at least two articles from the studies.
For RQ3, we will run another literature study (S3) to see the suggested metrics or practices to measure meaningful experiments. The result is expected to give us a mapping and better understanding of which metrics or practices should be used at a particular stage in an experiment. We aim to publish an article from the study.
These three studies should draw a better understanding of the challenges, inhibitors, and solutions in experimentation from a more holistic angle and how to measure the meaningfulness of an experiment. Based on this, we could identify which areas LLMs can be leveraged to make a meaningful impact. An immediate output of this is a prompt guidebook containing strategies and prompt templates (S4a). The guidebook will be tailored based on the knowledge found in the earlier studies. Later, a formal validation, in the form of case studies, will be conducted with selected software startups (S4b). We aim to publish at least one article from the study.
To address RQ5, we will further analyze the results of the earlier studies to identify which area of experimentation LLMs can support. Once decided, we will develop an LLM-based tool on top of one of the existing models. The tool will have the additional experiences from the earlier studies to generate relevant results (S5a). The tool can be further customized for specific needs through the few-shots approach. The users can also provide feedback in the form of relevancy level. The general concept of the tool can be seen in Figure 1. A fraction of the prompt guidebook may also be incorporated into the tool. Hence, these guides will be run automatically based on the prompt. Finally, a thorough evaluation and validation process will be together with selected software startups (S5b). The evaluation will be measured based on the users’ satisfaction level (identified in the first study). We expect to publish at least two articles from the study.
In this research project, initially, we will not set clear boundaries or criteria for the software startups we want to work with. This makes us open to any opportunity that might show up during the research. In the future, boundaries might be introduced when necessary.
The whole project activity in the design science research is shown in Figure 2. The first three studies aim to develop a holistic understanding of the problem, in this case, the experimentation in the context of software startups and the use of AI, especially LLMs, in the experimentation process. The last two studies have a diferent aim, which is to develop possible solutions and validate them internally. Lastly, the solution will be brought in the real context but in a limited scope.
This research project will run for five years. The detailed plan is shown in Table 1. The project started in early November 2023 and will run until November 2028. In the first year, we plan to run two studies, S1 and S2, to address RQ1 and RQ2. The studies are expected to be run in parallel. In the second year, we will do S3 to address RQ3. These three first studies are fundamental for the next two studies. Then, we will take the third year to run S4 to address RQ4. The remainder of the time will be used to run the final study, S5, and to address the last research question, RQ5. RQ1: What are the practices, challenges, inhibitors, and solutions in experimentation in software startups? RQ2: How do software startups incorporate AI technologies, especially LLMs, in new product development? RQ3: How to measure whether an experiment produces a meaningful result or not? RQ4: How to efectively use prompts to guide the experimentation process? RQ5: How to improve the experimentation process by leveraging the power of LLMs? Activity S1: Interview study with software startups S1: Interview study with software startups, S2: Literature study S3: Literature study S4a: Prompt guidebook dev. and S4b: Case study (for validation) S5a: AI assistant dev. and S5b: Case study (for validation)
Period 2023-2024 2023-2024 2025 2025-2026 2027-2028
There are at least five contributions from this research project, namely: • A better knowledge of how software startups tackle challenges and inhibitors during experimentation. This knowledge will help future research on developing new strategies to reduce, if not eliminate, the known inhibitors. • A better understanding of how far software startups have integrated AI technology into their experimentation pipeline. This will reveal which aspects of experimentation could be helped using the advancement of AI. • A guideline or a framework or a set of measurements to help software startups judge the meaningfulness of an experiment. • A validated prompt guidebook containing a guided walkthrough to help software startups do their experiments. With the help of additional contexts generated during the interaction, the walkthrough is expected to provide suggestions on which type of experiment the company should do and how to run the experiment. • An AI assistant tool personalized to the software startup and a specific project where the company will experiment. The tool is expected to be validated together with selected software startups.
This work has been supported by ELLIIT; the Swedish Strategic Research Area in IT and Mobile Communications. [18] L. Chen, M. Zaharia, J. Zou, How is ChatGPT’s behavior changing over time?, 2023. URL: https://arxiv.org/abs/2307.09009v2. [19] L. Sun, Z. Shi, Prompt Learning Under the Large Language Model, in: 2023 International Seminar on Computer Science and Engineering Technology (SCSET), 2023, pp. 288–291. URL: https://ieeexplore.ieee.org/document/10266544. doi:10.1109/SCSET58950.2023. 00070.