The interest in serious games (games that can educate, train, and motivate) has grown very fast in the last decade, yet the research community rarely studies serious game development. For instance, previous work has mainly investigated non-game developers' search practice and has proposed techniques and tools for general purpose software engineering. This paper reports on a pilot user study that was used to understand the thoughts and needs of (serious) game developers. We asked serious game developers “what”, “how”, and “why” they usually search online to develop their games. Our study reveals that software search for serious games development is mostly related to finding game-specific tools, documentation, and algorithms for AI, animation, rendering, and learning analytics. Also, specific web platforms (like web tutorials, Q&A sites, and public software repositories) are very relevant for serious games development. Finally, our analysis of serious game developers' search habit informs about the need for more advanced search engines with sophisticated query and filtering facilities.
But the efforts, time and costs of serious games development can be reduced if existing
software assets and other pedagogical components available on the market can be
effectively retrieved (or reused) and integrated into the development process
This study reports on a deeper exploration of serious games developers’ search practices. It was conducted as an online survey (using Google Drive) with professionals and practitioners in SG development. The main goal of this study was to explore the search habits of SG developers in order to understand the factors that can influence the way they search and reuse software and other related information. The survey was conducted using the following research questions: • RQ1: What software and other related information do you usually search to develop your serious games, and why? • RQ2: What are some approaches you usually use to FIND software and related information you need to develop your serious games? • RQ3: While searching online in the past, what are some obstacles that may have hindered your ability to FIND (RQ3.1) or REUSE (RQ3.2) what you needed to develop your serious game? • RQ4: Please describe one or more situations when you were trying to find a specific software or any software-related information on the Web…? Our research questions will be answered by interpreting their results. These questions will be indexed and discussed in the following chapter. 2 2.1
Nowadays, we observe a growing number of software engineers and researchers
focusing on SG development
Software Search is a very common practice in software engineering that aims at
discovering suitable software for a given purpose
Techniques used to study software search include web survey, interview, search
logs, and analysis of Q&A sites
To understand the specific needs and challenges of serious games developers, we
recently investigated a popular Q&A site (GameDev Exchange) to find out what SG
developers usually search to develop their games
This survey reports on a deeper exploration of serious games developers’ search practices. It was conducted as an online survey (created with Google Drive) with professionals, researchers, and students involved in the development of (serious) games. The study reports on the motivations of serious games developers seeking specific software and related information, how they go about finding help and which challenges they usually encounter to FIND and/or REUSE what they need. The study also informs about potential improvements in next-generation search engines. 3 3.1
Nowadays, there is a wide range of algorithms, tools, best practices, and frameworks available online that can be used to facilitate the development of full-scale serious games. To understand the challenges in identifying and reusing such tools and features, an online survey (developed with Google Drive) has been conducted. The main goal of this evaluation was to explore the search habits of serious games developers and to understand the factors that can affect the way they search and reuse software and other related information. So, we used our previously defined research questions RQ1-RQ4. Our research questions will be answered by interpreting their results. These questions will be indexed and discussed in the following chapter. 3.2
Different consortium partners took part in this survey. We advertised the survey to external gamification companies and educational institutions. Overall, this evaluation was attended by 40 people: 10 software developers (2 software architects, 7 programmers, 1 game designer), 6 scientists, 6 project managers (including 4 CEOs), and 18 students, all involved in the conception, design and implementation of serious games. 3.3
We used a web-based survey to conduct our evaluation because 1) information can be
gathered very easily and quickly from a wide audience, 2) the development effort is
very modest and reduced, 3) data can be analyzed quantitatively and qualitatively. The
survey consisted of a combination of standardized questionnaires (including free-text
and multiple-choice questions) and 1 open question about a specific search experience.
We designed the instrument of our analysis iteratively after analyzing previous surveys
targeting software search in computing
The evaluation was carried out from August to October 2018. We solicited responses using convenience sampling by sending an invitation to experts, researchers, and students from higher educational, all involved in the development of serious games.
Especially, we advertised the survey via e-mail distribution lists addressing different members of different members of the consortium. RAGE project partners (i.e. UCM, INESC, OUNL, UU, UPB, TUGRAZ). Applied gaming research and development actors: serious-games-solutions.de (The Gamification Expert), oztron.com (Serious Games and Simulations for Education Technology), kastanie-eins.de (Games and Learning), bible.com/kids (Apps and Games to teach the Bible to kids). Educational institutions: the German Institute for Games (Hochschule der Medien Stuttgart (HDM), ifg.hdm-stuttgart.de) and the Institute of computer science of the New York Institute of Technology (NYIT, nyit.edu).
The survey itself was conducted using the online survey platform Google Form. While more users trialled the system, in total 40 participants took part in this evaluation and provided their feedback via the online survey. 3.5
Demographics. The participants aged between 18 and 64. We asked participants
about their experience in serious games development: 56% had less than one year, 24%
had between 1 and 3 years of experience, and 20% had between 6 and 20 years of
experience. We also asked participants about their roles in their last serious games
projects: 46.7% worked as Software Developer, 43% worked as Software Architect
(Team Lead), 30% worked as Game Designer, and 10% worked as Test Analyst. The
number of years in programming and the roles of participants were routinely used in
past research to estimate the experience of developers
Participants were also asked to self-rate their search experience (Böhm et al. 2013). On a five-point Liker scale, 37.8% rated themselves as experts, 32.4% as experienced, 13.5% as moderately experienced, and 2.7% as inexperienced. Results show that in addition to the high level of familiarity in search practices, the group of respondents is characterized by a high degree of education, research and (serious) games development experience.
Search Motivation. To understand why serious games developers search specific software and related information, we asked: “What software and other related information do you usually search to develop your serious games, and why?”. The question contains a multiple choice box, allowing respondents to complete the list (Other). Other included “Research papers”. Figure 1 summarizes the results to this question.
Documentation how to use a specific game API Specific algorithms to track the player' interactions and learning outcomes
Code example to use as reference examples for animation, rendering, AI…
Solutions to fix bugs (security bugs, databases) Tools to analyze and check the game performance (e.g. Debugger, Profiler)
Third-party libraries to integrate in games (e.g. Networking, AI, Audio,…
Instructions to optimize databases and server performance (data… Script code to control the behavior of the game, animate objects, or… Best practices to design games for specific markets (e.g. educaitonal,…
Specific Game Engines to develop multi-platform games (etc. Unity,… Instructions about license terms (e.g. GNU GPL, LPGL) use open-source…
Guidelines how to release games on specific store (e.g. Google Play,…
API Experts' help to know how to install an API Stand-alone tools (e.g. application server, ERP package, database tool)
Public datasets or services (e.g. for testing newly implemented client…
We found that looking for documentation how to use specific game APIs (61.8% of respondents), specific algorithms (to assess players’ learning outcomes, increase their motivation, etc.) (58.8%), and code snippets to use as references examples (58.8%) were the most common motivations. Followed by solutions to fix bugs (50%), tools for analyzing and checking the performance of games (50%), and third-party libraries (50%).
Search Approach. To explore how serious games developers go about finding help, we asked respondents about the approaches they frequently use to find search software and related information, e.g. asking colleagues. Answer options were nonexclusive multiple choices, including “Other” option with a field for free-text input. Figure 2 provides a summary of our findings. 31 participants answered this question. The result shows that asking colleagues or fellow students (67.7% of participants), using general-purpose search engines (64.5%), visiting standard web pages (54.8%), and searching public software repositories (45.2%) were the most common approach used by the respondents. Others included social network resources (Youtube Tutorials) and specific web site (Asset Store). Figure 3 summarizes the most popular online web pages consulted by 33 respondents. Other include “developers.google.com/games”. In total 5 online sites have been named by the respondents, whereby GitHub (69% of respondents) and Stack Overflow (69%) are the most popular, followed by Asset Store sites (for unity and unreal game engines) (33%). This result shows evidence about the importance of social networks and online communities for serious games development. 25 (67.7%)
Asking colleagues or fellow students Searching using general-purpose search engines (e.g.
Google, Bing, Yahoo, Ask) Visiting standard web pages (web tutorials, blogs, etc.)
Searching public software repositories (Bitbucket,
SourceForge, GitHub, ProjectLocket, etc.) Searching using a code search engines (e.g. Google
Code Search, Koders, Krugle, Codase)
Looking in scientific literature Searching using standard software (Adobe Reader,
Microsoft Word, etc.) Searching my company repositories or knowledge
base Asking or searching mailing list archives or discussion forums
Challenges in Software Search. To understand the different challenges while seeking software and related information, we asked respondents what can prevent them to find (Figure 4) and reuse (Figure 5) what they found on the Internet. Respondents could select challenges in a multiple-choice box or extend the list with the “Other” option. Figure 4 reveals that 65.6% of our respondents had difficulty to find help online because of the following problems: requirement was too unique (65.6%) (which suggest that, in some special context, they might not know what exactly to search for), unable to locate close match (code snippet) to use as reference example (40.6%), wrong search queries formulated (37.5%), and too many alternative solutions to choose from (31.3%). Our survey also reveals that “poor formatted source code” (18.8%) could not prevent serious games developers to find software although this may affect the detection of reference examples (close match). Figure 5 summarizes the results about the challenges while seeking software for reuse. Incomplete functionality (59.4% of respondents), poor documentation (56.3%) were the most difficult challenges to software reuse. Followed by too much effort to integrate third-party libraries (43.8%), lack of testing instructions (40.6%), and incompatibility with the target system (40.6%).
My requirement was too
unique Unable to locate any close matching code snippet to…
Wrong search queries
formulated
Too many alternative solutions to choose from Lack of help by the online
community Too many options on
where to search Search queries too unique for my specific task Poor formatting of the piece of code sFoTFeIbihgNaisusDrtrcaqehwculiehen4sasg:ttitohyRonoaneotulsifmpnfneoeaerneyesiddnheeaasdtvhmetteouohltpidintpaehdlseveete,crlehqowdouphiecyyaseoottiubouraonrraxgeb“waiWlsmiittohhyemia?tlonee” fqoyFrtuoiegheueuestrrtmieeoxra5nety:louaRfhntfeeadesdverpereoidn“nfaOfasocemtrehsmudetlrota”iint.tphilAoeetnhncqeshyuwoeopiesuactrsieoftcobnhtuooo“xniWcRdweEhsiotaUnhwtSaetanErhreeaetdnIhsdonoeintmteeisorxoenncfoatleuwbltss”sailt.vroaTeetc.hfloeoAirssr additional slot for free text under “Other”. Answer total of 32 survey respondents answered this question. A total of choices were nonexclusive. A total of 32 survey 32 survey respondents answered this question.. respondents answered this question. A total of 32 survey respondents answered this question.
Search Case Experience. We wanted to learn more about how respondents seek software, so we sought examples of participants’ past experiences. We asked the following questions: “Please describe one or more situations when you were trying to find a specific software or any software-related information on the Web. What were you trying to find? How did you formulate your search queries? What approaches did you use? What problems did you have to find and/or reuse what you found? And how useful was the search result?” The survey form provided a text editing field where participants could write their responses in free-form text. We received a total of 12 responses, of which 8 contained substantial details about past search experience. Table 1 provides three examples taken from among those 8 responses. The analysis of these examples provides evidence that serious games developers are trying to overcome the permanent information overload found in existing search engines (like Google). They seek advanced search features such as filtering by a specific programming language (#User1), or how to optimize search queries by describing the search context with keywords (#User3). #User1 – Unity supports multiple script languages (js, C#) and ui options to do one thing. Often this is a problem with normal search engines like google because you can't filter for a specific language and get a lot of code snippets you can't use. For example when you want to add an gravityfield to an object in c# and search for "unity add gravityfield" you find a good answer in the unity forum for js but nothing for c#. #User 2 – I start with 2 /3 words. If no significant result i add another word . Problem: too many sponsored responses within the top results #User3 – Once I searched for a tool that could generate JavaScript code for a node server. My main search terms where "swagger", "node", "code gen". Unfortunately the search engine just returned a lot of bullshit like a small project called "swagger-node-codegen" (written in JavaScript). Nothing really helpful for me and my purpose. After several days of investigation, I found a code generation tool, written in Java, which also produces/generates JavaScript (NodeJS-Server) code.
I think, the main problem was, that the search terms I used where to "generic" for this specific search request and even in different conjunction there are too much "possibilities" about what I could have needed. In other words - I was not able to describe my requirements in a unique and distinct search request. What I missed was the possibility to describe my context! For example, that I need the resulted code to be JavaScript, not that the generator is written in JavaScript.
Information Desire about Software. To help inform the development of advanced search tools, we sought to determine what kind of features our respondents would like to see implemented in future search systems. We posed the following question to all participants “What issue(s) do you think needs to be addressed in existing search mechanisms and/or tools to support serious game development?” (Figure 6). The question was in form of a multiple choice box with an “Other” option as free text. As shown in Figure 6 “Outdated exclusion” and “Content context-sensitive search” were added to the Other option. The analysis of responses to this question reveals that there is a need for more sophisticated filtering and query features that are well integrated with existing search engines, are easy to use and can optimize search results based on the user’s context more efficiently.
More research in the effectiveness of search
engines More options for excluding unnecessary search
results Making advanced search features easily
accessible for less experienced users
More public funding of research and projects More options for ranking search results based on the current search context
Other - Outdated exclusion Other - Content context sensitive search 1 1 10 12 13 14 16
Overall, the feedback gathered in this study shows that specific online tools (like search engines, public software repositories, and Q&A sites) are relevant for (serious) games development, because they can provide different kind of help (instructions, algorithms, and tools) that can facilitate and accelerate the development of serious games. Also, this survey provides evidence about the need for more advanced search engines with sophisticated query and filtering facilities.
This research was partially funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 644187, the RAGE project (www.rageproject.eu). 12