=Paper=
{{Paper
|id=Vol-3316/industry-paper2
|storemode=property
|title=The Experimentation Machine: A Framework to Support Assumption-driven Entrepreneurship in Startups
|pdfUrl=https://ceur-ws.org/Vol-3316/industry-paper2.pdf
|volume=Vol-3316
|authors=Harshil Paliwal,Slinger Jansen,Sjaak Brinkkemper
|dblpUrl=https://dblp.org/rec/conf/icsob/PaliwalJB22
}}
==The Experimentation Machine: A Framework to Support Assumption-driven Entrepreneurship in Startups==
https://ceur-ws.org/Vol-3316/industry-paper2.pdf
The Experimentation Machine: A framework to
support assumption-driven entrepreneurship in
startups
Harshil Paliwal1 , Slinger Jansen2 and Sjaak Brinkkemper3
1
Utrecht University, Utrecht, The Netherlands
2
Lappeenranta University of Technology, Finland
3
Utrecht University, Utrecht, The Netherlands
Abstract
Startups have a high rate of failure and they fail because entrepreneurs invest their resources based on
poorly tested assumptions. This is a waste of costly time and resources. In this research, a framework
named the Experimentation Machine is developed that helps entrepreneurs to adopt assumption-driven
entrepreneurship. If entrepreneurs use the Experimentation Machine, it is expected that they become
more familiar with assumption-driven entrepreneurship, and possibly even more successful as startups.
The framework has been evaluated with seven startups, who used the Experimentation Machine over a
period of 10 weeks. Our findings confirm that startups benefit from the framework and that it enabled
them to quicker unearth (incorrect) assumptions.
Keywords
Assumption-driven entrepreneurship, Startups, Startup validation, the experimentation machine
1. Introduction
Whenever an entrepreneur finds a scalable solution and decides to build a business around it, it
leads to a startup. There are numerous intricacies to understand when one takes a deeper dive
in this realm of startups. This essentially includes, the various techniques that the entrepreneurs
use to solve the problems, scale their solutions, and different methods that act as a guide to get
them through uncertainties. Needless to mention that a startup does not always succeed. There
are various unknown factors, apart from motivation, that contributes to its organic growth [1],
and one of them is testing the right assumptions.
We define IPAs, i.e., Innovation Process Assumptions, or assumptions for short, as simple
statements that are falsifiable, valuable, and testable.It should not be ambiguous or lead to
multiple solutions. This format is inspired from the works of [2]
Entrepreneurs often make assumptions towards a solution and customer requirements. To
test assumptions at an early stage is one of the several ways to mitigate risks [3, 4]. A similar
approach was introduced by Eric Ries in his book named The Lean Startup [5]. Ries talks about
ICSOB ’22 : 13th International Conference on Software Business, Nov 8–12, 2022, Bolzano, Italy
Envelope-Open harshilpaliwal@gmail.com (H. Paliwal); slinger.jansen@uu.nl (S. Jansen); s.brinkkemper@uu.nl
(S. Brinkkemper)
Orcid 0000-0003-3752-2868 (S. Jansen); 0000-0002-2977-8911 (S. Brinkkemper)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
a build-measure-learn loop that promotes experimentation to build a successful startup. Similar
views are echoed by [6] in their work that introduces a prototype-centric learning model. A
recent study was conducted to understand how these assumptions are engineered in software
startups [7]. But the problem is that despite the abundance of information that is available on
the application of lean principles, experimentation, or prototyping, practitioners still find it
difficult to implement them [8].
Studies conducted by [9, 10] reflect that poor business model is a cause of startup failure.
This means that the process of identifying the problem, developing a solution, identifying the
customer, making the customer aware of the solution, and delivering the solution, is somewhere
broken. First, the entrepreneurs must identify the correct problem. Without this, a startup is
creating something that the founders believe will help people but that is, in fact, not true. Second,
it is of utmost importance to understand what the customer needs before starting the product
development. Meaning, the solution might be unique but there is a chance that it will be too
expensive or unavailable or complex. Third, it should be ensured that founders have identified
their early adopters. Without early adopters there will be no feedback on the minimum viable
product and thus it will be impossible to understand if the startup is succeeding [9]. Another
factor is that, even if the founders have a perfect business model, they eventually start focusing
on a part of it [10]. Either the focus is too much on developing the product, or too much on
marketing without having a reliable product. The continued focus on the overall business model
is the essence of this research. Assumption-driven approach has been received well and has
the potential to guide founders in the startup journey. There is still a need for a framework
that could be easily understood and adapted. The goal of this research is to encourage the
adoption of the assumption-driven entrepreneurship in startups by constructing, and validating
a framework.
2. The Experimentation Machine
One of the main reasons for startup failure is a failing business model. But there is so much
more to a startup than just making the right business plans. The founders need to make
crucial decisions almost every day. On top of it, if they want to adopt assumption-driven
entrepreneurship, they need to dedicate extra time towards planning, prioritising, and testing
the assumptions. In this section, we are describing the framework named The Experimentation
Machine. The aim is to provide a detailed guide and a simple tool to the founders to practice
assumption-driven entrepreneurship within the startup routine. The Experimentation machine
framework is visualised in the figure 1. Detailed description of the elements is as follows:
Assumption: This element is an essential part of the Experimentation Machine. It is present
to establish the practice of assumption-driven entrepreneurship using this framework.
Template: The definition of an assumption should be simple, testable, and falsifiable. Inspired
from the works of [2], the assumptions for this framework are one line statements in the
following format:
As a [role] I state that [statement].
Each assumption also has a role attached to it. The phrase used to support this is “As a [role]”.
Figure 1: The Experimentation Machine
This helps with understanding its setting. For example, if the assumption is related to revenue,
then the role will be the chief financial officer. Some examples are:
As a co-founder I state that I have a landing page for users.
As a technical lead I state that the team has required technical skill set to create an MVP.
As a co-founder I state that the startup will be accelerator ready in the next six months.
These examples provide an idea on how to write simple assumptions using the provided
template. Every assumption can always be elaborated to cover the granular details by adding a
description to it using the collaboration tool.
Lifecycle: Each assumption goes through various stages before it is accepted or rejected.
There is a flow to it and this is shown in the figure 2. An explanation of each of the lifecycle
stages is as follows:
Figure 2: Lifecycle of an Assumption in the Experimentation Machine
First, assumptions are Defined by brainstorming with the team about their assumptions, using
examples of assumption for inspiration. An assumption can be Delayed if the team decided to
include an assumption, but has not made any progress on it. An assumption can also be Altered
when it over time proves to be unsuitable. If the assumption is not helping the startup team as
planned, it can be Discarded.
However, most assumptions are Planned for further inspection and research. At this phase
the team assigns the assumption to one of its members and adds further descriptions to it,
such as a timeline. The team then sits together and defines the steps that need to be taken in
the Experiment Design. In practice, the experiment design consists of the definition of several
steps that can be checked off. After the design, the Experiment Execution takes place. If the
experiment goes well, the assumption is Rejected or Accepted.
Assumptions and Startup Lifecycle: According to [11] every assumption can be fit into a
context, such as a Problem, Solution, or Customer. For the Experimentation Machine framework,
the assumptions are inspired from the the startup lifecycle and the business model canvas [12].
Most of the assumptions are defined keeping in mind the maturity of a startup. There are some
assumption which are “repeating”, meaning they could be completed but will need to be tested
again after a certain time period. Here is a list of assumptions that have been pre-defined based
on the startup lifecycle:
Stage - Ideation/Problem-Solution Fit
• As a [role] I state that we will be incubator ready till [Date].
• As a [role] I state that I know my early adopters.
• As a [role] I state that I have a good overview of the competition.
• As a [role] I state that I know how the target customer will use my product.
• As a [role] I state that I know my product is needed in the market.
Stage - MVP
• As a [role] I state that I know what my MVP is.
• As a [role] I state that the team has all the required technical knowledge to build an MVP.
• As a [role] I state that I do not need any funds to build an MVP.
• As a [role] I have a complete interview protocol to reach my early adopters.
• As a [role] I state that I have a plan for MVP testing.
• As a [role] I state that I have a plan for MVP release.
Stage - Product-Market Fit
• As a [role] I state that I have a pricing model for the product.
• As a [role] I state that I know my potential customer.
• As a [role] I state that we will be incubator ready till [Date].
• As a [role] I state that there is a book keeping system in the company.
Stage - Scale-Up
• As a [role] I state that I have funds for customer acquisition.
• As a [role] I state that I have a social media strategy.
Repeating assumptions
• As a [role] I state that my team is complete.
• As a [role] I state that the company has enough funding till [Date].
• As a [role] I state that the introduction deck to pitch the startup is complete.
• As a [role] I state that I know the key metrics for my startup.
These assumptions can be used as it is and are a good starting for the founders with just an
idea. As the company grows, the assumptions change with it and hence can be customised.
Collaboration Tool: To use the Experimentation Machine, the team needs a collaboration
tool. It can be any software application that allows them to work together. For this research we
are using Trello1 . Following is a step-by-step guide to set up the Trello board for a startup and
how to regularly update it.
Figure 3: Collaboration tool - after the research
1. Start with a clean board. This can be done using the “Create Board” option. Trello
provides the option to add and remove members on a board. Members can have different
levels of access on the board.
2. Create four “Lists” in the board. Name the list as follows - Backlog, In Progress, Complete,
Future Assumption.
3. Create 12 “tags”. 9 tags are for each category of the Business Model canvas, they all have
different colors. The other 3 tags are defined, planned, and delayed. They all have black
colors.
4. Create a “Card” for every assumption.
a) In the beginning, all the cards should be placed in the Backlog list.
b) Assign BMC category tags to each card. Assign “Defined” tag to each card. Each
card can have multiple BMC tags.
c) Open the card to enter the description. Write details about the assumption including
- current status and acceptance criteria.
5. Move a card from Backlog to In Progress.
a) Remove the tag “Defined”
b) Assign the tag “Planned”
c) Open the card and add following details - a checklist describing the experiment
steps, and update the current status.
1
https://trello.com
d) Assign people to the card.
e) Provide a date of completion.
f) If the assumption has not been tested within the given time then assign the tag
“Delayed”.
6. Move the card from Backlog to Future Assumption if the team decides to test it at a later
stage. Assign date to the card. The idea is, at the assigned date Trello will notify the team
about the assumption which should be tested. If the assigned date has passed, discuss
and assign a new date (this does not mean that the assumption is “Delayed”) or move it
to In Progress.
7. Move the card from In Progress to Complete if the assumption has been accepted.
8. Move the card from In Progress to Complete if the assumption has been Rejected. In this
case, edit the card and add the word “REJECTED” in the beginning.
The tags in the Trello help categorise the assumption into BMC sections. This provides a
complete picture to the teams about the startup. The different colors of the tags provide a
diversity within the board thus helping to visualize that the team is in fact looking at their
startup from every perspective. For example, let’s assume that the colour of tags “Key Activities”
and “Customer Segment” is yellow and green respectively. Now if there is a lot of yellow on the
board and not even a hint of green, it might be an indication that the team is too attached to
their product/idea. They are investing a lot of time building/improving the product without un-
derstanding the right customer segment. Multiple categories to the assumptions can be assigned
if required. For example - “As a [role] I state that the startup will be incubator/accelerator ready
by [date].” Can be categorised as “Key Activities” (because it will help with startup growth) and
“Key Partners” (because incubators/accelerators are external facilitators to a startup). Another
example is - “As a [role], I state that I have received positive customer feedback about the MVP.”
can be categorised as “Value Proposition” (as it helps the founders confirm if they are creating
value) and “Customer Relationship” (because positive feedback supports customer retention).
Thus, we use these labels to help segregate the assumptions and ensure that all areas of a startup
are the point of focus.
Within the set up guide it is advised to write down the experiment in the form of a checklist.
This helps identify small atomic tasks that are easy to perform, eventually leading to the final
result. Trello allows creation of multiple checklist within a card. This can be used for not only
listing down the experiment steps but to also jot down sub-assumption if possible. If an card is
dependent on a different card then the checklist can include task like - “Complete card B before
moving forward.” This helps to manage the dependency between assumptions.
3. Case Study Analysis
During the research a total of 182 assumptions were defined with the 7 startups who participated
in the case study. Out of the 182, 112 assumptions were tested. This means that during the
period of 10 weeks, 112 assumptions were either in the “Plan”, “Delay”, “Experiment Design”,
“Experiment Execution”, “Accepted”, or “Rejected” stage. We continue with a detailed analysis
of the assumptions providing insight into how startups uniquely practise assumption-driven
entrepreneurship.
The assumption - “As a co-founder, I state that the team has the required technical knowledge to
build an MVP”, was quickly tested by all the startups. Many startup founders come up with ideas
that are outside their domain. For example, EM-E wanted to provide an alternative to milk
in a carton. To come up with the initial version of the product, they had to rely on external
factors. The founders must understand if their team has the required technical capabilities to
build an MVP. This research revealed that founders were aware of this gap/availability within
their startup.
The assumption - “As a co-founder, I state that I have a pricing model for the product”, took a long
time to test. This was the scenario for all the startups. They kept pushing it to the “Future” list or
took at least two months to decide on an initial model. A large part of creating a pricing model
depends on factors like - knowing the target market, studying the competition, and understand
your unique selling point. Understandably, startups in their early stages will find it difficult to
test this assumption. More mature startups also found it difficult because their company is still
evolving with the customer’s needs. They are still trying to find the right product-market fit.
The assumption - “As a co-founder, I state that I know what my MVP is”, was the most volatile
assumption. There is a very valid reason why every company faced some difficulties before
reaching a positive test result. It is difficult for a team to narrow down a startup idea into an
MVP. Every founder wants to output the best product into the market so that customers like it
and provide positive feedback. However, this is never recommended. One should always get a
minimal product out and then improve it (or add features) based on the customer review and
not the other way round. Thus, for this assumption, every team struggled with the test results.
They kept discussing the possible functionalities, starting from scratch and gauging the market.
Thus, the assumption kept getting delayed, altered, or rejected. Let’s take a deeper dive into
this assumption and see how it was tested by startup EM-E.
MVP started in the “Defined” stage. Within this stage it was decided that the assumption will
have following to-dos:
• Decide product name.
• Decide product size (this was dependent on the external vendors).
• Decide product pricing (this was dependent on production cost, competitor analysis).
This was necessary because once the product is built, it cannot be stored and should be
sold immediately.
After one week of discussions, the assumption was moved to “In Progress”.
It took one week to finalise the product name.
It took one month to finish the other two items in the to-do list. After this the assumption was
moved to “Complete”.
The entire assumption took one and a half months to complete.
The assumption - “As a co-founder, I state that I know my early adopter”, was tested differently
by almost every startup. This is once again a very obvious observation. Early adopter depends
on the type of product that the startup is building, or the customer segment that the startup
wants to enter. And to know early adopters means reaching out to them and ensuring that they
will be the first ones to test the MVP and provide feedback. For EM-G, this was easy because
their idea can be tested by friends and family. For EM-E, and EM-A this was very difficult. EM-E
needed a large number of early adopters because their target market was the direct consumer.
They ran various social media campaigns to achieve this. EM-A had to reach out to businesses
and convince them to use their product as a plug-in for existing services. The founders tried to
exploit their professional network as much as possible to get meetings with the companies.
The assumption - “As a co-founder, I know the key metrics for my startup”, was pushed to the
“Future” column by all the startups. This is concerning. It is understandable to pinpoint the key
factors that determine the success of an idea is difficult. But the sooner startups decide this, is
better. Key metrics help them focus on what is important, know what works, and find out the
point of improvements. This shows that founders tend to focus on taking actions - build an
MVP, set up an interview with customers, find vendors, etc. But they spend less time focusing
on details that can help them measure their growth.
The assumption - “As a co-founder, I state that I have a good overview of the competition”, had the
same testing method for almost all the startups. The testing method usually involved performing
market research of the competitors, understanding their marketing strategy, pricing strategy,
etc. Some startups also performed a gap analysis to learn their unique selling point.
The list helps understand how the Experimentation Machine can be adopter in a unique by
every startup. The various testing method that can be applied to each assumption shows the
diversity among the case studies and the participants. Furthermore, because every founder has
a different priority, the assumptions are tested at different rates in various orders.
3.1. Test technique Analysis
Every startup has their own way of testing the assumption. Table 1 provides an overview of
various testing techniques that was utilised by startups:
From the table 1, one can observe some set techniques that startups use for particular assump-
tions. One of the most prominent ones is collecting feedback during the product-market fit
phase. For this, there are various assumptions that the startups made use of: “As a co-founder, I
state that I have a plan to collect customer feedback”, “As a co-founder, I know how to analyse
customer feedback”, “As a co-founder, I know that my product is needed in the market”, “As a
co-founder, I know that the potential customer is aware of my product.”, and more. All of these
assumptions are confirmed only when the founders are interacting with the customers in some
way. However, during the study, an observation was that founders show slight hesitance to
reach out and speak to the customers.
This is quite prominent in “younger” startups. EM-B, EM-D, and EM-G pushed the related
assumptions to the “Future” column for the entire duration of the research. EM-A started
approaching customers only towards the end of the research that led them to rethink their
product idea. However, the hesitation does not always stem from the possibility of negative
feedback. Many times, it is not easy to reach out and directly interact with the end customers.
In the case of EM-E, just to get the first 100 customers to land on their website, required them
to spend a chunk of money on online ads. Another reason for this is the founder’s attachment
to the product. They keep trying to perfect the MVP/product. EM-D had access to their
potential customers through their professional network. But they always approached this step
as something that you perform after you have finished everything. EM-G also had access to
Table 1
Various testing techniques used by startups
Startup Lifecycle phase Assumptions Techniques
Create product map
Problem-solution fit Deciding MVP Assess internal technical capabilities
Literature Study
Customer feedback (Interviews)
Customer feedback (Questionnaire)
Problem-solution fit Testing MVP Concierge Service
Usability Test
Taste Test
Gap Analysis
Problem-solution fit Competitor Analysis
Market Research
Marketing emails
Product-market fit MVP Release Social Media campaigns
Directly speaking to the customer
Social media campaigns
Reaching the early
Product-market fit Professional network
adopters
Personal Network
Interviews
Product-market fit Collecting feedback
Questionnaires
Market research
Product-market fit Pricing strategy
Competitor Analysis
the customers, but in their case, the unclear business model hampered the progress. They had
no clear idea about the type of product they wanted to build. It took them two sessions to
figure out a concrete product plan and, after that one month, they began approaching the target
market. Reaching out to the customer requires effort. But, this is the step that should be taken
early on to ensure that the startup will create value.
3.2. Startup Analysis
During the case studies, every participating progressed differently. This was due to various
factors like team dynamics, hours spent on startup, type of assumptions tested, etc. Table 2
demonstrates how the startups progressed through out their case study participation.
Based on the data gathered during the case studies, the following insights are drawn:
Startups EM-A and EM-E made the most progress during the case studies. They made
ample use of the Experimentation Machine and together defined and tested the most number
of assumptions. This helped them make tremendous headway. EM-A had a team of three
co-founders. When EM-A started with us, their startup was hardly a month old. The team had
excellent dynamics, conducted healthy and fruitful discussions during the sessions, and made
consistent efforts to grow their idea. As a result, within a period of 10 weeks, they successfully
registered their company, entered an incubator validation program, had their first version of
MVP, and exploited their professional network to the fullest to get customer feedback and
secure early adopters. The feedback helped them improve their MVP more and more, as well as
rework some bits.
Table 2
Startup progress during the research.
Type of
Startup Before Research After Research Active
assumptions tested
- Problem-Solution Fit phase. - Developed first version of MVP. 4 MVP
- No MVP. - Interviewed with various companies 4 Customer
EM-A - No customer interaction. and finally found an early adopter. 7 Revenue/Cost Yes
- No formal agreement between - Joined incubation program. 4 Partner
founders. - Formalised company registration. 4 Resources
- Problem-Solution Fit phase. - Developed first version of MVP. 4 MVP
- No MVP. - Performed market research and com- 7 Customer
EM-B - No customer interaction. petitor analysis. 7 Revenue/Cost No
- No formal agreement between the - Team cannot spend hours and hence 7 Partner
founders. discontinued it. 4 Resources
7 MVP
- Launched crowdfunding campaign
- Product-Market Fit phase. 4 Customer
with pre-orders of the new product.
EM-C - Assessing market to launch the next 4 Revenue/Cost Yes
- Joined an accelerator program.
product. 4 Partner
- Needed to hire new resources.
4 Resources
- Problem-Solution Fit phase. - MVP almost ready. 4 MVP
- MVP in progress. - Worked on protocols to reach out to 4 Customer
EM-D - No customer interaction. customers. 4 Revenue/Cost Yes
- No formal agreement between the - Hired new resources. 7 Partner
founders. - Identified startup USP. 4 Resources
- Product-Market Fit phase 4 MVP
- Problem-Solution Fit phase. - Launched with their first product. 4 Customer
EM-E - No MVP. - Finalised and launched the marketing 4 Revenue/Cost Yes
- No customer interaction. strategy. 4 Partner
- Became part of an incubator. 4 Resources
- Ready to apply for research grants by
4 MVP
- Product-Market Fit phase. partnering with external agencies.
4 Customer
- Struggling with resourcing decisions. - Took huge strides in improving the ex-
EM-F 4 Revenue/Cost Yes
- Struggling with funding. isting MVP.
4 Partner
- Did not have a fixed target market. - Ready to hire resources.
4 Resources
- Have a fixed target market.
4 MVP
- Problem-Solution Fit phase. - Released an MVP as a concierge ser-
4 Customer
- No business model. vice.
EM-G 7 Revenue/Cost No
- No MVP. - Customer interaction and feedback
7 Partner
- Internal team conflicts. - Decided that the idea will not work.
4 Resources
EM-E is a team of two co-founders who were working part-time on their idea. They used
the Experimentation Machine as a method to get themselves organised and test the biggest
assumptions about the target customer. For them, getting to know the best product-market
fit was the key. They tested every possible customer segment and only after that launched
their social media strategy. Eventually, they found the best partners for their product, the right
market, and their brand image. This helped them have a successful first launch.
During the research, we worked with two startups that eventually closed down. Those
startups are EM-B and EM-G. There are several ways in with the Experimentation Machine
helped them realise some limitations. EM-B had a dedicated team of five, and they initiated their
idea only a few weeks before participating in the research. They were in the ideation phase. As
seen in the table 2, they did not test assumptions from a lot of different categories. They mostly
tried to build their MVP and performed some feasibility tests. They passively tried to reach out
to the target market but quickly dropped the assumptions and pushed it to the future column.
It is not that the MVP did not work, but the co-founders understood that they could not invest
their time into this idea because of prior commitments. The variety and amount of assumptions
helped them take a step back and understand that building a startup is so much more than just
building a product.
For EM-G, the reason was different. They had a good team of two who wanted to work
part-time on the idea. However, they did not have a good business model and this is widely
known as the most common reason for startup failures. They did not know what their MVP
should be. To figure this out, they came up with a concierge service that will help them test
the feasibility of their idea. Even though the results of this test was somewhat encouraging,
they also led to more unnoticed issues. To launch a good product, the founders needed to get a
team member who knows the field (and perform an in-depth domain study ). This led to some
frustrations that started to affect the team dynamics. Twice they changed the steering team of
the company and eventually, after four weeks, decide to shut down the project.
EM-C and EM-F were mature startups. It means that they were older than at least one year
and were already making money. It helped experiment with different and custom assumptions
with them. All the other startups were still in the problem-solution fit stage, and most of
the assumptions they used were already “Defined”. For example, find early adopters, collect
customer feedback, release the MVP, and more. But the mature startups already knew these
things about their company. Their main focus was to improve their existing product and grow
the customer base. Thus, they defined custom assumptions based on their current situations
such as - fresh social media strategy they wanted to experiment with, or a new feature to add to
the existing product and understand the customer reaction, or try and engage with an entirely
new customer segment. Some examples of it are: “As a co-founder, I state that feature [x]
will help increase the ease of usage of the product.”, “As a co-founder, I state that I know my
potential market.”, “as a co-founder, I state that the upcoming crowdfunding campaign will help
increase Instagram reach by [n].” Another thing that was observed during the literature study
and the case study is that the co-founding team tends to undergo some differences of opinions
when the company starts growing. EM-F founders were not able to have a similar vision for
the future. Assumption - “As a co-founder, I state that my team is complete” was repetitive for
both the mature startups. This is because they kept increasing their team by hiring interns
or full-time employees. The assumption was moved to “Define” for both of them after being
marked as “Complete”. It shows that startups need to grow their team as the company evolves.
The analyses helped understand the diversity the case studies. It shows how every startup is
different. This is causes by difference in founder’s mindsets, ideas, speed of growth, so many
more factors.
4. Evaluation
Case studies were conducted with seven startups to introduce and validate the Experimentation
Machine. To introduce the framework, five sessions (each two hours long) were set up with
each startup. A questionnaire was prepared, inspired from [13], to evaluate the Experimentation
Machine. After completing the research, the participants were asked to evaluate the framework
by answering questions. Listed below are the key finding obtained after the evaluation and an
analysis of the research validity.
The Experimentation Machine is easy to use. During the feedback interviews five startups
said that the framework is easy to use. The other two said that it is easy once you get into the
practice of using it. Also, the collaboration tool, Trello, was appreciated by the participants.
The entire process of creating cards, assigning people, setting timelines, and assigning BMC
categories truly helped them with their startup idea in a holistic way. After the end of five
sessions, they were confident that they could setup the collaboration tool entirely on their own
according to the guidelines of the Experimentation Machine.
The Experimentation Machine helps to adopt assumption-driven entrepreneurship. All
the participants agreed that the framework helped them understand and practice assumption-
driven entrepreneurship. During the sessions, all the participants were encouraged to share
their assumptions about the startup. All of them agreed that during every session, it was only
about putting down their assumptions into relevant assumption that helped their startup. Six
startups said that after the end of all the five sessions they had completely grasped the idea
behind the framework. The seventh startup dropped out midway because the founders decided
not to continue with the idea.
Writing assumption for startups needs practice. Almost all the feedback answers said that
it was not easy to write the assumptions unless you practice the technique. It is tricky to
understand the entire idea of writing down your assumptions into the assumption format
provided. This is a new practice for all the participants. Furthermore, due to action-oriented
environment within the startup culture, it is difficult to think of translating these actions into
assumptions.
Founders find the idea of assumption-driven entrepreneurship new. This concept has
been around for a while and we have covered this in detail in our literature study. However,
this findings suggest that the concept is still new to the founders. Five out of seven startups
said that they did not know about assumption-driven entrepreneurship before participating in
this research. Two said that they had an idea of it but never really understood how to apply it
in their startup. All the startups said that they had never implemented this practice in their
startup routine. They also mentioned that they have never worked with anything similar to the
Experimentation Machine before.
Assumption-driven entrepreneurship helps startups. All the participants agreed that the
Experimentation Machine helped their startups. One of the startups dropped out midway be-
cause the founders decided not to continue with the idea. This was startup EM-B. However, even
for this case study there was a positive confirmation for the framework and the entire concept.
The founders of EM-B agreed that the Experimentation Machine helped them understand that they
should not work on their startup. Thus, the practice of writing and testing assumptions made
them realise their assumptions were wrong. It helped them take a step back without investing
a lot of resources into their startup. This is the aim of assumption-driven entrepreneurship
and this framework - to test the assumptions, understand what will work, and invest resources
accordingly.
The findings provide insight into the way the research was conducted and the positive
outcomes of it. This is quite encouraging and open the door to many directions in which the
research can grow if the time permits. This is explored more in the last chapter.
Table 3
Summary of the Experimentation Machine Framework Evaluation Results
Case Evidence
Evaluation Prominent
4 Strongly supported
Characteristic Comments
7 Fairly/Strongly opposed
4 “Did not know about the concept before.”
4 12
Background 7 “Knew about the concept because we kind of followed it (divide and
72
conquer strategy) but did not put it in such a formalised way.”
4 “The framework helped the startup find direction and footing.”
“Now I know the concept even better.”
4 30
Goal 7 “Don’t know yet if we will use it regularly. Do see the value but
75
not sure on how well we will be able to maintain. It is always hard to
implement a new thing in the routine.”
4 “Yes the framework is easy to use.” “It will be even more beneficial
4 35 if the team is bigger.”
Environment
77 7 “Writing assumptions is not very straightforward.” “Because it is a
new way of thinking you do nee some practice to get into it.”
4 “Yes the framework has all the elements to support assumption-
4 30 driven entrepreneurship.”
Structure
75 7 “Defined, delayed, planned tags are a bit much and not needed be-
cause assigning the dates does the job.”
4 “Yes the number of sessions were enough to grasp the concept.” “Yes
4 56 the sessions were conducted in a planned and structured manner.”
Activity
77 7 “It was enough but would like more help with actual assumption
writing.” “In the early stage the sessions should be every week.”
4 “Yes. Would love to read a book/paper about the framework.“ ”Yes
4 18 we know how to accept or reject assumptions.”
Evolution
72 7 “Will not read anything because we are already familiar with the
concept. Good to have as a reference.”
5. Conclusion and Future Work
The objective that was set out for this study was to encourage the adoption of assumption-driven
entrepreneurship in startups. The concept has been around for quite some time but there was a
lack of a framework that can guide the founders during the adoption process. The preceding
chapters reported on the design of the Experimentation Machine. It is a conceptual framework
that aimed to capture the dimensions and layers of this concept to introduce it to the founders
and guide them. The strength of the Experimentation Machine lies in the fact that it tries
to combine a startups routine to the business model canvas by defining assumptions. This
ensures that the founders are constantly looking at all the areas. The visualisation of it using
the collaboration tool integrates deadlines, and makes task assignment easier.
This research has the potential to branch out and take various directions. One of the areas to
explore is the follow-up with every startup. In the future, after the set number of sessions (five),
every startup should be asked to start using the framework without the supervision of coaches.
Then after a cooling period, the researchers and participants should come together to discuss
the usage of the framework. This can potentially provide novel insights into how startups work
and adopt new practices. Furthermore, this increases the use duration of the Experimentation
Machine in a startup.
References
[1] R. van Cann, S. Jansen, S. Brinkkemper, Software business start-up memories: Key decisions
in success stories, Springer, 2012.
[2] J. Melegati, X. Wang, Quest: new practices to represent hypotheses in experiment-driven
software development, in: Proceedings of the 2nd ACM SIGSOFT International Workshop
on Software-Intensive Business: Start-ups, Platforms, and Ecosystems, Penguin Books Ltd,
2019, pp. 13–18.
[3] M. Gutbrod, J. Münch, M. Tichy, How do software startups approach experimentation?
empirical results from a qualitative interview study, in: International Conference on
Product-Focused Software Process Improvement, Springer, 2017, pp. 297–304.
[4] M. Gutbrod, J. Münch, M. Tichy, The business experiments navigator (ben), in: 2018 IEEE
International Conference on Engineering, Technology and Innovation, IEEE, 2018, pp. 1–8.
[5] E. Ries, The lean startup, Crown Business, 2011.
[6] A. Nguyen-Duc, X. Wang, P. Abrahamsson, What influences the speed of prototyping? an
empirical investigation of twenty software startups, in: International Conference on Agile
Software Development, Springer, Cham, 2017, pp. 20–36.
[7] J. Melegati, E. Guerra, X. Wang, Understanding hypotheses engineering in software
startups through a gray literature review, Information and Software Technology (2020).
[8] J. Bosch, H. H. Olsson, J. Björk, J. Ljungblad, The early stage software startup develop-
ment model: a framework for operationalizing lean principles in software startups, in:
International Conference on Lean Enterprise Software and Systems, Springer, 2013, pp.
1–15.
[9] L. Mikle, Startups and reasons for their failure, SHS Web of Conferences 83 (2020).
[10] M. Cantamessa, V. Gatteschi, G. Perboli, M. Rosano, Startups’ roads to failure, Sustainability
(2018).
[11] J. Rai, S. Knoll, PoSoFiHy-A guided approach towards formulation of problem solution fit
hypothesis (2018).
[12] A. Osterwalder, Y. Pigneur, Business model generation: a handbook for visionaries, game
changers, and challengers, John Wiley & Sons, 2010.
[13] N. Prat, I. Comyn-Wattiau, J. Akoka, A taxonomy of evaluation methods for information
systems artifacts, Journal of Management Information Systems 32 (2015) 229–267.