This paper reports the difficulties of recruiting participants at the client's site in an empirical project on requirements engineering & management. The author discusses the origins of the problems and some stated reasons for the participation. These results lead in a short discussion about the improvement of empirical research on the requirements engineering & management process at the client's site.
Known studies on software project processes seem to be mostly conducted on the producer's site, especially in cooperation with software producing companies. The reason is obvious: Software projects are a core business for software producing companies, who are interested in improving their processes. But business information systems are systems whose success is related to the usage by the clients. Therefore research at the client's site should be important. On the client's site the research focus seems to be the (business) success of information systems, according to the DeLone & McLean information systems success model (Urbach et al. 2008; Urbach et al. 2009).
Research on processes on the client's site is rare and is concerned from special influencing factors. This paper will show and discuss some problems, observed in an empirical study that was conducted in 2010/2011 in Germany in companies outside the IS business (Weißbach, R. 2013). The paper starts with a short description of the study and the recruitment of participants (Chapter 2), followed by a chapter on the stated difficulties for participation (Chapter 3) and a chapter on triggers for participation (Chapter 4).
Chapter 5 presents and discusses the conclusions. Chapter 6 will show some ideas for further work.
Acknowledgements. I like to thank the reviewers of this paper for their valuable advice to work out the ideas more precisely.
The Study and the Recruitment of Participants But the research on RE&M focuses on the main actors in software engineering: requirements engineers, project managers, developers. Business department staff is commonly seen as object in the requirements elicitation process. The active participation of business department staff in contrast is disregarded in literature and obviously ignored in research (checked against the summarization in (Cheng, B., Atlee. J. 2007).
In this situation the author wanted to conduct 25 semi-structured personal interviews in the area of Hamburg, Germany, as a pilot study. The project was staffed by the author and a student assistant. Five participants had been found by personal contacts. To get the other 20 participants we collaborated with a regional entrepreneurs' association ("Bundesverband Mittelständische Wirtschaft", Hamburg). We thought to contact 50 member companies by personal telephone calls to get a relevant, but not ex ante quantified number of participants and added a call for participation on the website of this association. To clarify: The interviews should be conducted personally, the telephone calls should only be used to arrange the interviews. In return, the participants have been announced to get the results of the study and to get an invitation to a free workshop on RE.
No company accepted the personal invitation and only two companies responded to the presentation on the website. But both of these companies had been software companies, who were interested in the result of the study.
Therefore we decided in 2011 to make personal telephone calls by the student assistant to make appointments. The student assistance had experience in acquiring participants for marketing research studies. We picked telephone numbers from public telephone directories and asked for responsible persons in IS and/or business departments. The telephone agent worked from the university's site, so that the university's official telephone number was transmitted. To get 18 participating companies it was necessary to contact ca. 900 companies (multiple calls counted only as one contact), equivalent to a response rate of 2%. This response rate seems to be very low, but we did not find information about response rates in comparable situations. Typical re-Introduction Many companies mentioned the lack of time or a privacy policy as reasons to their non-attendance. But these have not been the only arguments. Focusing on the main research topic, we did not record and count the answers explicitly at that time. Therefore the following aspects should be seen as indicators, not as clear and complete results.
Companies in the IT branch are interested in market research and in improving processes. For these companies the benefit of participating in research projects is obvious. But what could be the interest for companies in other branches to participate in IT research projects?
The focus on IS research at the user's site is the success in IT projects. This topic is an accepted research topic (DeLone and McLean, see Urbach et al. 2008, Urbach et al. 2009). While this topic addresses the management in general, research on RE&M processes is a very specialized topic. Referring to the "rigor vs. relevance" discussion (Benbasat, I., Zmud, R. 1999, Lyytinen, K. 1999) it seems that RE&M is not relevant for management.
Research Topic is a "Non-Issue"
Many of the asked companies told us, that they are not interested in the research topic, because it is not relevant to them. This argument could be interpreted in different directions:
Unknown vocabulary: The term "requirements engineering" (or the German translation, "Anforderungsanalyse") is not known. Anticipating this danger, the telephone agent paraphrased the problem additionally. Lack of awareness: The importance of this topic is not realized. -Or: The importance of this topic is realized, but it is no problem in praxis. -Or: The topic seems to be not relevant, because IS are not seen as important for the core processes of the business.
Empirical Research has no direct benefit to the participants: One reason is that there is no direct output of the research action. The other is "difference in timeframes of action between academics and practitioners" (Kuechler, W., Vaishnavi, V.. 2011: p. 127). A typical result of empirical research is a benchmark that could be useful for the participants. But this benchmark refers to a former situation that must not be valid any more.
In Design Science Research (Vaishnavi, V.,Kuechler, W. 2012) the benefit for the participant is more concrete and immediate.
Triggers for Participation
Some people agreed to participate because they remembered their own university background. They wanted to support the university in general or the scientists and they wanted to get back in contact to the university to discuss and reflect their positions. This argument was produced by participants in companies with a relative low proportion of academic staff.
Some people had been interested in the research topic due to their professional education. These people were computer specialists, programmer, technicians, regardless of an academic degree.
Some participants acquired awareness of RE&M by own experiences in projects. Most of them reported problems in the project due to poorly conducted RE&M, including a complete project failure. Other participants realized early enough the importance of RE&M in larger projects.
Some participants had been interested in the reflection and discussion of their practice, as a kind of consulting. They were interested in the results of the research for improving their own knowledge.
Mouth-to-Mouth References / "Snowball Principle"
One participant gave us the phone number of a colleague who was interested in this research topic, too.
Starting the project we thought that the effort for recruiting participants would be less.
According to (Benbasat, I., Zmud, R. 1999) who specified applicability (= utility), currency and interest to professionals as important, we thought to have a research topic well fitting to the companies' needs. But it seems that research at the client's site is more difficult than at the producer's site, because the benefit for the participants will often not be directly recognizable. Also, results of empirical research will need longer time for dissemination than the design of artifacts.
Due to the lowly estimated importance of empirical research on RE&M processes at the client's site, it seems to be difficult to establish new research directions. Therefore we will depend on qualitative researche.g. case studies, grounded theoryto understand the diversity in RE&M processes especially on the client's site.
This paper analyses an RE&M research project with an untypical research question and a heterogeneous group of potential interviewees. Therefore the observations could not be seen as valid for other research questions in Software Engineering in general.
The response rate for typical research questions with a homogeneous group of potential interviewees will be higher.
Regarding the research experiences described in this paper and the preliminary results described in Weißbach, R. (2013), we will state a lack of understanding of internal processes and of collaboration processes between internal and external staff on the client's site in the RE&M process. To work out a framework for how RE&M processes are conducted at the client's site, grounded theory and case studies will be the first valuable approach. This framework could be enhanced with quantitative empirical research.