=Paper=
{{Paper
|id=Vol-2584/PT-paper6
|storemode=property
|title=How to Gather Requirements from the Crowd with Hackathons – Challenges, Experiences and Opportunities
|pdfUrl=https://ceur-ws.org/Vol-2584/PT-paper6.pdf
|volume=Vol-2584
|authors=Patrick Mennig,Frank Elberzhager
|dblpUrl=https://dblp.org/rec/conf/refsq/MennigE20
}}
==How to Gather Requirements from the Crowd with Hackathons – Challenges, Experiences and Opportunities==
https://ceur-ws.org/Vol-2584/PT-paper6.pdf
How to Gather Requirements from the Crowd with
Hackathons – Challenges, Experiences and Opportunities
Patrick Mennig Frank Elberzhager
Fraunhofer IESE, Fraunhofer-Platz 1, 67663 Kaiserslautern, Germany
{patrick.mennig;frank.elberzhager}@iese.fraunhofer.de
Abstract
[Context and motivation] Today’s software systems become more
and more complex, especially when we think about connected sys-
tems such as cyber-physical systems or digital ecosystems. Customers
thereby demand flawless apps and have several needs in mind that
such solutions should provide. If these are not fulfilled, they do not
use the solution. In such connected environments, usually many dif-
ferent stakeholders exist that all have different requirements. [Ques-
tion/problem] In complex cyber-physical systems, the manifold re-
quirements and possible solutions can overstrain requirements engi-
neers and developers. What are ways to consider needs and require-
ments from different stakeholders? How can such input be used for re-
quirements engineering? [Principal ideas/results] In order to gather
ideas, issues and requirements from several different stakeholders, we
propose to consider hackathons besides well-known and established re-
quirements engineering methods. With these, one gets to know his
stakeholders, get real needs from his stakeholders, and get early ideas
and prototypical solutions. [Contribution] We share our experiences
with two hackathons we performed in a research project that aims at
developing a climate neutral city district with supporting digital ser-
vices. We discuss opportunities and challenges and how results might
be used for requirements engineering.
1 Context and Motivation
The world we live in today is becoming increasingly connected. Connected cars, households and factories,
intelligent and mobile devices and even combinations of these are just a few examples. This trend runs through
both the private life and business sectors and today is often referred to as digital transformation. For companies
there are new opportunities to develop new or offer their products and services in such connected systems, and
the chances for innovative products and new businesses are increasing. In this context, terms such as Internet
of Things (IoT), digital ecosystems or cyber-physical systems have become established.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International
(CC BY 4.0).
In: M. Sabetzadeh, A. Vogelsang, S. Abualhaija, M. Borg, F. Dalpiaz, M. Daneva, N. Fernández, X. Franch, D. Fucci, V.
Gervasi, E. Groen, R. Guizzardi, A. Herrmann, J. Horkoff, L. Mich, A. Perini, A. Susi (eds.): Joint Proceedings of REFSQ-2020
Workshops, Doctoral Symposium, Live Studies Track, and Poster Track, Pisa, Italy, 24-03-2020, published at http://ceur-ws.org
While such connected systems offer enormous opportunities for companies, there are also many challenges in
the development and operation of such systems. The development of embedded systems or information systems
as such is often already characterized by high complexity, but their integration into highly connected systems
increases this complexity considerably. Challenges are e.g., the high heterogeneity, organizational questions
like opposing motivation of the actors, evolutionary aspects or contradictory quality requirements. In addition
to these challenges, the pressure to be fast on the market is increasing, i.e. a goal for companies is often to
achieve a short time-to-market in order to remain competitive. Furthermore, quality plays a decisive role in user
acceptance, but also in the innovative power and thus the sustainability of such systems.
From the perspective of requirements engineering (RE), one big question is what are the “right” requirements.
While this was always a relevant question that a requirements engineer asks their customers, the number of stake-
holders in the sketched connected systems is much higher. Therefore, additional ways of gathering requirements
from different stakeholders become more relevant [2].
Currently, we are working in the “EnStadt:Pfaff” 1 research project which goal is to build a climate-neutral
Smart City district with the help of digital solutions. There, several stakeholders such as citizens, companies,
research institutions, business companies have to be considered. From our perspective, different ways of involving
such stakeholders are needed to provide digital solutions that really offer a benefit and fulfill the needs of the
different stakeholders. Living labs and hackathons are means for citizen involvement [1].
One way we started with, involving people outside our project partners, was performing hackathons. A
hackathon is a hacking marathon where a group of people develops some kind of prototypes in a short amount
of time [3]. Such hackathons exist since the early 2000. From our perspective, they are not used broadly,
and rather on conferences as some kind of additional event or by bigger companies mainly for recruiting and
marketing goals. Applying these hackathons to involve of stakeholders, especially citizens, allows interested
stakeholders to participate in elicitation and solution finding activities. Apart from social and business benefits,
technical benefits are expected [4]. This raises the question how the results of the hackathon can be used for
further RE activities. Organizing public hackathons is time consuming and costly, however many stakeholders
can participate at once. Making use of the results in an efficient manner is crucial for considering them successful.
From a research perspective, we are considering two questions:
• How can the results from a hackathon be efficiently used for RE?
• How suitable are hackathons for bringing together “usual people” with limited IT background but relevant
issues, requirements, and questions together with the more technic affine requirements engineers, developers
and product owners, in the RE process?
Until now, we performed two hackathons in the last two years in the given research project. We got great input
for our research project content-wise. Furthermore, we involved several people (more than 80) and therefore got
ideas (and prototypes) that are relevant for different stakeholders in the future.
In this contribution, we share experiences we made during the planning, execution and follow-up activities
of the hackathons. We further discuss the two research questions and provide an outlook on further ideas
for hackathons. With this contribution, we want to motivate other researchers and companies to start with
hackathons to get further input for their requirements engineering with the overall goal to develop services and
apps that fit the needs of the stakeholders even better. In addition, we want to discuss with other researchers
how results from a hackathon can be better included in requirements engineering activities.
2 Organization of Hackathons
In the following section, we explain our approaches for planning, executing and following up the two hackathons,
enabling the readers to understand and replicate the approach. The first hackathon was performed in June 2018,
the second one in October 2019. While many aspects are the same or similar, there were also some differences
between the two hackathons, based on our experiences from the first.
2.1 Planning
Main topics to consider during the planning phase were: Finance, location, technology equipment, marketing,
registration, t-shirts, catering, event schedule and follow-up activities. Figure 1 shows an overview of work
1 https://pfaff-reallabor.de
Management Financing Marketing Location Schedule Follow-Up
Kick-Off Person Days E-Mail Booking Registration Questionnaires
Efforts
Regular Posters Furniture Pre-Event Feedback
Meetings Catering Information Sessions
Flyers Electricity Distribution
Planning Team Prizes Follow-Up
Organization Personal Wifi General Meetings
Giveaways Marketing Information
Supervisor Additional Capturing of
Management T-Shirts Social Media Hardware Topics Results
Goals
Partners Networking Recreation Area
Sponsors Event
Website Decoration Schedule
Projection
Signs
Figure 1: Activity plan for the organization of hackathons.
Hackathon 1 Hackathon 2
Number of participants more than 30, no special experience re- more than 50, targeted specifically at
quired participants with at least some develop-
ment experience
Teams Pre-assigned to teams by organizers By choice of participants
Focus of results New ideas, new scenarios, paper proto- Connected scenarios, prototypes
types
Time Friday, 4 pm — Saturday, 5 pm Friday, 2 pm — Saturday, 3 pm
Schedule Registration (1 h) Registration (1 h)
Introduction and topics (1 h) Introduction (0.75 h)
Ideation (5.5 h) Ideation, prototyping and development
Intermediate pitch (0.5 h) (21.25 h)
Prototyping and development (15.5 h) Final presentations and award ceremony
Final presentations and award ceremony (2 h)
(1.5 h)
Table 1: Overview of main differences between the first and second hackathon.
packages and activities required to perform during the planning stage. Both the planning of marketing activities
and event schedule have been crucial activities for the shape of the hackathons, while follow-up activities are
important for the usage of the results within the project context. Inviting stakeholders to participate in such
type of event requires serious investment in marketing activities. We have reached out via posters, flyers (e.g.,
at universities, public library, town hall) email marketing, social media campaigns and through our professional
network. For the first hackathon, we started planning around three months before the event. Due to our
limited experience with this type of event in our organization, the schedule was ambitious. The planning of the
second iteration began around eight months before the event, hence being a lot less demanding, thus it could be
integrated better with other project work.
2.2 Execution
Table 1 shows a comparison of differences in the shape of the first and second hackathon. Our hackathons2
were 24-hour events (except registration). We started Friday 5 pm (respectively 3 pm at the second hackathon)
2 Pictures taken during the hackathons can be found under https://pfaffhack.iese.de/pfaffhacks.
and ended the next day. Officially, the participants could arrive one hour earlier to get their info package and
material such as badges and t-shirts.
At the time of the first hackathon, the research project was in a phase where the focus was on finding general
ideas and scenarios how to support different stakeholders with digital services in the city district. Therefore, we
motivated the participants to think into this direction. We presented the project, its scope and challenges and
provides a set of general topics that the teams could select to further elaborate on. Afterwards, we guided them
with design activities such as brainstorming and paper prototyping to come to initial ideas that they could refine
during the night and the morning. During the night, some participants left and came back the next morning,
which had some influence on the development of the results. During the latter morning, the teams began to start
with their presentations that were about 15 minutes. Three teams presented their results, and the solutions were
judged by a jury of three experts. Due to the small number of remaining participants, all teams got some prices.
During the second hackathon, we focused much more on elaborating concrete ideas and ensured that the
participants had at least some basic programming skills. The introduction was shorter and the teams started
with elaborating ideas after 45 minutes of describing the project and the goals. Consequently, they had much
more time to come to concrete solutions, but were also more responsible on their own. Supervisors though
supported to the different groups with general and technical advice and reflected the progress together. In
addition, some technical intros were given by the supervisors regarding solutions and technology the teams could
use, especially about the technical platform prototype provided. This is a simplified version of the system that
is used in the “EnStadt:Pfaff” project to flatten the required learning curve. As we had eleven teams during the
second hackathon which were mainly working on solutions (i.e. also during night, most people stayed awake), the
teams only had four minutes time for the presentation the next day. These were prepared during the morning
by the teams, and presented at 1 pm, Saturday. The best three teams, again judged by a jury of three experts
(providing technical, business and smart city expertise) received some prices.
Besides all working times, we had some breaks in between: Experts gave short inspiring talks about technolo-
gies and topics related to the climate neutral smart city district. Short fitness sessions were held to energize the
participants. Meals were provided in the evening, the night, the morning and for lunch. Tired participants could
use a dedicated area with comfortable seats for short recreation.
2.3 Follow-Up
Immediately after the events, the supervisors have tidied up the location and captured any results. Paper
prototypes, sketches, paper cards with ideas and the like were photographed to conserve them for later use. At
the first hackathon, the teams’ presentations were recorded (video and audio). Due to organizational problems,
we only took pictures of the presenting teams and their solutions at the second hackathon. Shortly after both
events, review meetings with all supervisors and relevant project members were held. The initial reaction was
the same for both hackathons: all agreed that the events’ results provided valuable benefits for the project, the
publicity works were improved and pursuing another iteration is worthwhile. The required effort for the planning
and execution was mentioned as a negative aspect of the hackathon event format.
3 Discussion and Experiences
The planning and execution of the first iteration of the hackathon was challenging due to short time until the
event and lack of experience. In order to incorporate the hackathon’s results into the project, no later date could
be chosen. We have strong experience with regular project work and other participation formats, but the open
nature and long duration of the event required special attention. For the planning of the second hackathon we
could rely on the experiences from the first, hence starting earlier, extending the available time and allowing us
to focus on improvements. It shall be noted that the planning of the second hackathon was significantly simpler.
Organizations seeking to hold their own hackathons might reach out to experienced partners or plan for building
their own body of experience. This type of event might be more suitable when aiming for several iterations.
Most important of all was that we see both hackathons as very successful events, meaning a hackathon is
indeed a good way to bring people with a different background together (though, of course, not the only way).
Due to the local nature of the given research project, stakeholders have a strong interest in the (preliminary)
project results and the overall developments. The hackathons provided a way of integrating citizens into the
project, enabling them to provide requirements, ideas and solution concepts by themselves. Strong interest in
the project goals, hence the hackathons’ goals, might be an influencing factor for their participation. As the
events were held over 24 hours, participants that signed up had a strong internal motivation to contribute with
valuable input. Due to the relaxed nature of the hackathons, participants and supervisors had a lot of fun. This
is an aspect that sometimes falls short in other participation formats.
During both hackathons, after the teams have pitched their results to the jury, three winning teams were
chosen and awarded prizes. These were given by partners of the hackathon (e.g., vouchers for an exit game). No
cash prize was given to any participant. Still, their motivation to pitch the solution was high and the winning
teams showed joy and pride after receiving a prize. We believe that the cash equivalent of the prizes is less
important to the participants than the honor of actually receiving it.
Legal aspects need to be considered when planning and conducting such an event. General terms and condi-
tions for the usage of participants’ data, photographs and the results produced need to be prepared and agreed
upon by all participants. It shall also be noted that occupational health and security regulations apply that re-
strict the availability of supervisors during the event (e.g., by German law, employees are not allowed to regularly
work for more than ten hours per day and rest periods need to be adhered to). This can be circumnavigated
either by special agreements or thorough planning of working schedules.
The results were integrated into the concepts developed in the “EnStadt:Pfaff” project. For us as the executing
organization, the hackathons provided an opportunity to integrate supervisors and project members from various
departments in the planning and execution, leading to a strong increase in communication about the project and
its results.
Regarding the first research question, we have some open questions remaining. During both hackathons, a
total of 14 teams created individual results that were presented in front of the whole audience and the jury. In
short time, a large number of results is presented that were previously created over 24 hours. The results created
during a hackathon are by far neither created by requirements engineers nor by professional software engineers.
They are focused on meeting the hackathon goal and producing tangible as well as likeable output rather than
having their documentation in mind. Recording the presentations conserves at least the core essence of the
solutions, but many intermediate results from the thought processes participants go through are lost. This also
holds true when keeping access to source code and other artefacts created during the hackathons, as participants
typically focus on creating solutions rather than writing down supporting information. It remains unclear, how
participants can be supported in documentation efforts without this leading to cognitive overload.
In the first hackathon, the focus was much broader than in the second. This resulted in a much more diverse
background and age distribution of participants in the first. During the night, many participants left the event
location to return the next day. We assume this was due to less resistance to sleep deprivation or family
obligations (due to the overall higher age of participants). Participants of the seconds hackathon were younger
and their background was less diverse (mainly computer science students). We assume that this led to mostly all
of them staying overnight and working all the time. This aligned with the intended audience, as we advertised
the second hackathon to participants with, at least, some development experience due to the focus of results.
Shorter event schedules or two-day events with a night-break seem to align better with expectations of a more
general public audience. However, the second hackathon led to results that better aligned with the project goals
and intended use of a software ecosystem for smart-cities. One needs to carefully plan, advertise, schedule and
execute (i.e. design) the event for the intended audience and expected type of results. Hackathons seem to be
suitable event formats for incorporating general citizens and experienced stakeholders alike.
In the two hackathons performed, we had to consolidate the results afterwards. Of the results of the first
hackathon, many ideas were also already thought of before in the project by the researchers, but some new ideas
came from the participants of the hackathons. This changed during the second hackathon, where many new
ideas emerged (maybe simply due to the higher number of participants). However, the initial goal of the second
hackathon, namely to create more connected solutions, did not work out very well, as the teams had enough to
do with creating and realizing their concrete services. This is again up to us researchers and project partners to
base new solutions on the results of the participants, hence incorporating the vision of a connected ecosystem
more strongly into the design of the hackathon.
Comparing the results of the first and the second hackathon, the sheer number of teams, hence individual
results, differs (three vs. eleven). What is still open to us which and how to exactly transfer the many ideas into
concrete services. This is one of our future tasks.
4 Summary and Future Work
In this poster paper, we presented our experience with two hackathons we planned, organized and performed
in scope of the “EnStadt:Pfaff” project. The goal was to generate initial ideas and prototypical solutions for
digital services within a climate-neutral city district. Our main research questions thereby focused on how to
get information and requirements from a diverse crowd in that environment.
We believe and experienced that hackathons are a great source for new ideas and initial prototypes for software
services. Though they do not fit for all purposes and in all situations, hackathons can often be used to gather and
explore concrete requirements and to evaluate early implementations of services and apps. The two hackathons
conducted were beneficial to the overall project goal and the involvement of stakeholders in the requirements
engineering and development activities. Hence, we plan to continue this type of an event on an annual basis, at
minimum during the remaining project duration.
Acknowledgements.
Parts of this work have been funded by the “EnStadt: Pfaff” project (grants no. 03SBE112D and 03SBE112G)
of the German Federal Ministry for Economic Affairs and Energy (BMWi) and the German Federal Ministry of
Education and Research (BMBF). We thank Sonnhild Namingha for proofreading.
References
[1] Baccarne, B., Mechant, P., Schuurman, D., Colpaert, P., De Marez, L.: Urban socio-technical innovations
with and by citizens. Interdiscip. Stud. J. 3(4), 143–156 (2014)
[2] Doerr, J., Hess, A., Koch, M.: RE and society - A perspective on RE in times of smart cities
and smart rural areas. Proc. - 2018 IEEE 26th Int. Requir. Eng. Conf. RE 2018 pp. 100–111 (2018).
https://doi.org/10.1109/RE.2018.00020
[3] Komssi, M., Pichlis, D., Raatikainen, M., Kindstrom, K., Jarvinen, J.: What are Hackathons for? IEEE
Softw. 32(5), 60–67 (2015). https://doi.org/10.1109/MS.2014.78
[4] Valença, G., Lacerda, N., Rebelo, M.E., Alves, C., de Souza, C.R.B.: On the Benefits of Corporate Hackathons
for Software Ecosystems – A Systematic Mapping Study. pp. 367–382 (2019). https://doi.org/10.1007/978-
3-030-35333-9_27, http://link.springer.com/10.1007/978-3-030-35333-9_27