This paper reports the difficulties of recruiting participants at the client's site in an empirical project on requirements engineering & management. The author discusses the origins of the problems and some stated reasons for the participation. These results lead in a short discussion about the improvement of empirical research on the requirements engineering & management process at the client's site.
Known studies on software project processes seem to be mostly conducted on the
producer’s site, especially in cooperation with software producing companies. The
reason is obvious: Software projects are a core business for software producing
companies, who are interested in improving their processes. But business information
systems are systems whose success is related to the usage by the clients. Therefore
research at the client’s site should be important. On the client’s site the research focus
seems to be the (business) success of information systems, according to the DeLone
& McLean information systems success model
This paper will show and discuss some problems, observed in an empirical study
that was conducted in 2010/2011 in Germany in companies outside the IS business
Acknowledgements. I like to thank the reviewers of this paper for their valuable advice to work out the ideas more precisely. 2
In 2009/10, the author started a study on the participation of business department staff in the requirements engineering & management [RE&M] process. The aim of this study was to get a more differentiated view on the RE process in business information systems projects and in non-project work. The study was focused on (but not exclusively limited to) small and medium enterprises [SME].
RE&M in general is a topic that is covered by many textbooks and empirical
research. The importance of the RE&M process for the success (or the failure) of
projects is generally accepted in literature
In this situation the author wanted to conduct 25 semi-structured personal interviews in the area of Hamburg, Germany, as a pilot study. The project was staffed by the author and a student assistant. Five participants had been found by personal contacts. To get the other 20 participants we collaborated with a regional entrepreneurs’ association (“Bundesverband Mittelständische Wirtschaft”, Hamburg). We thought to contact 50 member companies by personal telephone calls to get a relevant, but not ex ante quantified number of participants and added a call for participation on the website of this association. To clarify: The interviews should be conducted personally, the telephone calls should only be used to arrange the interviews. In return, the participants have been announced to get the results of the study and to get an invitation to a free workshop on RE.
No company accepted the personal invitation and only two companies responded to the presentation on the website. But both of these companies had been software companies, who were interested in the result of the study.
Therefore we decided in 2011 to make personal telephone calls by the student assistant to make appointments. The student assistance had experience in acquiring participants for marketing research studies. We picked telephone numbers from public telephone directories and asked for responsible persons in IS and/or business departments. The telephone agent worked from the university’s site, so that the university’s official telephone number was transmitted. To get 18 participating companies it was necessary to contact ca. 900 companies (multiple calls counted only as one contact), equivalent to a response rate of 2%. This response rate seems to be very low, but we did not find information about response rates in comparable situations. Typical response rates for telephone interviews with companies in Germany are 20-30% (Koll, C. 2006)
In 4 of the 25 participating companies, the interviews had been conducted simultaneously with 2 interviewees working in the same company, either in the same or in different departments. 3 3.1
Many companies mentioned the lack of time or a privacy policy as reasons to their non-attendance. But these have not been the only arguments. Focusing on the main research topic, we did not record and count the answers explicitly at that time. Therefore the following aspects should be seen as indicators, not as clear and complete results. 3.2
Companies in the IT branch are interested in market research and in improving processes. For these companies the benefit of participating in research projects is obvious. But what could be the interest for companies in other branches to participate in IT research projects?
The focus on IS research at the user’s site is the success in IT projects. This topic is
an accepted research topic
Many of the asked companies told us, that they are not interested in the research topic, because it is not relevant to them.
This argument could be interpreted in different directions: Unknown vocabulary: The term “requirements engineering” (or the German translation, “Anforderungsanalyse”) is not known. Anticipating this danger, the telephone agent paraphrased the problem additionally.
Lack of awareness: The importance of this topic is not realized. - Or: The importance of this topic is realized, but it is no problem in praxis. - Or: The topic seems to be not relevant, because IS are not seen as important for the core processes of the business. 3.4
Empirical Research has no direct benefit to the participants: One reason is that there is
no direct output of the research action. The other is “difference in timeframes of
action between academics and practitioners”
In Design Science Research
Some people agreed to participate because they remembered their own university background. They wanted to support the university in general or the scientists and they wanted to get back in contact to the university to discuss and reflect their positions.
This argument was produced by participants in companies with a relative low proportion of academic staff. 4.2
Some people had been interested in the research topic due to their professional education. These people were computer specialists, programmer, technicians, regardless of an academic degree. 4.3
Some participants acquired awareness of RE&M by own experiences in projects. Most of them reported problems in the project due to poorly conducted RE&M, including a complete project failure. Other participants realized early enough the importance of RE&M in larger projects. 4.4
Some participants had been interested in the reflection and discussion of their practice, as a kind of consulting. They were interested in the results of the research for improving their own knowledge.
One participant gave us the phone number of a colleague who was interested in this research topic, too. 5
Starting the project we thought that the effort for recruiting participants would be less.
According to
But it seems that research at the client’s site is more difficult than at the producer’s site, because the benefit for the participants will often not be directly recognizable. Also, results of empirical research will need longer time for dissemination than the design of artifacts.
Due to the lowly estimated importance of empirical research on RE&M processes at the client’s site, it seems to be difficult to establish new research directions. Therefore we will depend on qualitative research – e.g. case studies, grounded theory – to understand the diversity in RE&M processes especially on the client’s site. This paper analyses an RE&M research project with an untypical research question and a heterogeneous group of potential interviewees. Therefore the observations could not be seen as valid for other research questions in Software Engineering in general.
The response rate for typical research questions with a homogeneous group of potential interviewees will be higher.
Regarding the research experiences described in this paper and the preliminary results described in Weißbach, R. (2013), we will state a lack of understanding of internal processes and of collaboration processes between internal and external staff on the client’s site in the RE&M process. To work out a framework for how RE&M processes are conducted at the client’s site, grounded theory and case studies will be the first valuable approach. This framework could be enhanced with quantitative empirical research. 8