1 Baselining Wireless Internet Service Development - An Experience Report Fabio Bella, Jürgen Münch, and Alexis Ocampo Abstract — New, emerging domains such as the engineering of wireless Internet services are characterized by a lack of experience based on quantitative data. Systematic tracking and observation of representative pilot projects can be seen as one means to capture experience, get valuable insight into a new domain, and build initial baselines. This helps to improve the planning of real development projects in business units. This article describes an approach to capture software development experience for the wireless Internet services domain by conducting and observing a series of case studies in the field. Initial baselines concerning effort distribution from the development of two wireless Internet pilot services are presented. Furthermore, major domain-specific risk factors are discussed based on the results of project retrospectives conducted with the developers of the services. Index Terms — Effort Quality Models, Process-centric Knowledge Management, Risk Factors, Wireless Internet Services. —————————— ‹ —————————— 1 INTRODUCTION T he engineering of wireless Internet services is an emerging application domain characterized by quickly evolving technology, upcoming new devices, new improve the software development process as applied within the observed organizations; GQM-based measure- ment is practiced to gather quantitative experience, communication protocols, support for new, different types whereas qualitative aspects are addressed by the retrospec- of media, and varying and limited communication band- tive-based collection of lessons learned. Therefore, this width, together with the need for new business models that study should be seen as a challenging attempt to character- will fit in with completely new service portfolios. Examples ize a promising new application domain not only from a of new wireless Internet services can be expected in the qualitative, but also from a quantitative point of view. Of domains of mobile entertainment, telemedicine, travel ser- particular interest is the focus placed on first effort distribu- vices, tracking and monitoring services, or mobile trading tion baselines gathered from the development of suitable services. pilot services. Due to its recentness, this domain lacks explicit experi- Section 2 introduces the methodologies applied within ence related to technologies, techniques, and suitable soft- the study and explains how they relate to it. Section 3 dis- ware development process models that is based on quanti- cusses the context in which the case studies were per- tative data. Unreliable project planning, incorrect effort formed; the overall approach applied to gather quantitative estimates, and high risk with respect to process, resource, as well as qualitative experience, and the results in terms of and technology planning, as well as with regard to the effort distribution baselines and major domain-specific quality of the resulting product are inevitable consequences risks observed. Section 4 subsumes the article and sketches of this lack of experience. One means to capture experience future work to be performed. and get valuable insight into a new domain is systematic tracking and observation of representative pilot projects. 2 BACKGROUND This paper presents a study consisting of two case stud- ies aimed at quantitative baselining. Additionally, the case This study is based on a combination of descriptive process studies were used to gain qualitative experience. The article modeling [4], GQM-based measurement [5], [15], and retro- aims at giving managers and developers a sense of the be- spective-based collection of lessons learned [7]. This Section havior of projects in the wireless Internet domain. gives an overview of the methodologies applied and ex- The approach followed in this study is based on a com- plains how they relate to the study. bination of descriptive process modeling, GQM-based The main idea of the descriptive process modeling is to measurement, and collection of lessons learned: Descriptive explicitly document the development processes as they are process modeling is applied in order to understand and applied within a given organization: A so-called process engineer observes, describes, and analyzes the software ———————————————— development process and its related activities, and provides • F. Bella is with the Fraunhofer Institute for Experimental Software Engi- descriptions of the processes to the process performers. neering (IESE), Sauerwiesen 6, 67661, Kaiserslautern, Germany. E-mail: Since the processes are usually complex, support is needed bella@iese.fraunhofer.de. • J. Münch is with the Fraunhofer Institute for Experimental Software Engi- for both process engineers and process performers. De- neering (IESE), Sauerwiesen 6, 67661, Kaiserslautern, Germany. E-mail: scriptive process modeling is applied within the context of muench@iese.fraunhofer.de. the study with the help of the Spearmint® environment. • A. Ocampo is with the Fraunhofer Institute for Experimental Software Engineering (IESE), Sauerwiesen 6, 67661, Kaiserslautern, Germany. E- The architecture of Spearmint® and its features for a flexi- mail: ocampo@iese.fraunhofer.de. ble definition of views, used for retrieving filtered and tai- 2 QUATIC’2004 PROCEEDINGS lored presentations of process models, is presented in [4]. The duration of the project is 30 months and an iterative, One distinct Web-based view, namely the Electronic Proc- incremental development style is applied: three iterations ess Guide (EPG), is used for disseminating process informa- are performed, of roughly 9 months each. tion and guiding process performers, e.g., project managers In iteration 1, a first version of the planned pilot services and developers. was built using GPRS. At the same time, a first version of The Goal/Question/Metric (GQM) approach is applied methods and tools was developed. to define measurement goals and a proper measurement In iteration 2, a richer second version of the pilots was infrastructure [5], [15]. During the first two steps, business developed on GPRS, using the first version of methods and and improvement goals are analyzed and metrics defined tools. In parallel, an improved second version of methods according to the process model elicited through Spear- and tools was developed. mint®The results of this first phase are GQM plans that In iteration 3, the final version of the pilots is being de- comprise all metrics defined. veloped on UMTS, using methods and tools from the sec- In the following step, the project plan and the process ond iteration. Also, a final version of methods and tools is model are used to determine by whom, when, and how being developed. data are to be collected according to the metrics. The data Currently, the third iteration is still running. The case collection procedures are the results of this instrumenta- studies discussed in this article refer to data from the first tion. two iterations. Raw data are collected according to the data collection procedures. The collected raw data are analyzed and inter- 3.2 Case Studies Method preted according to the GQM plan and the feedback pro- This subsection presents the process-centric approach ap- vided by the interested parties. plied within the WISE project for gathering experience in In the next step, the interested parties draw conse- the new domain. quences based on the analysis and their interpretations. As mentioned in the previous subsection, in parallel to Finally, analysis, interpretations, and consequences are the development of the pilot services, a measurement infra- resumed in the measurement results and collected as ex- structure was defined in order to evaluate the effects of the perience in the experience database for future reuse. method and tools applied to develop these services. The In addition to the measurement of quantitative data, the infrastructure is based not only on measures but also on collection of qualitative data is driven by project retrospec- interviews and any other available evidence. tives [7]. Therefore, meetings and interviews with the par- Figure 1 sketches the strategy applied iteratively during ticipants of different work packages are conducted regu- each of the three iterations to gather, package, and maintain larly to focus lessons learned and improvement potentials. experience from the development of the pilot services. The Concerning more specific wireless-related topics, published experience acquisition process is depicted using the Spear- experience in the field were important sources of informa- mint® notation: The circles represent activities, the rectan- tion, particularly at the beginning of the study. An exten- gles artifacts, the arrows indicate produces/consumes rela- sive overview of related work is given by [9]. tionships between activities and products. 3 CHARACTERIZING EFFORT IN THE WIRELESS INTERNET SERVICES ENGINEERING DOMAIN In the following, the context of the study, the approach ap- plied, and the main related results are described. 3.1 Context of the Case Studies The present study was conceived as an integral part of the evaluation of the Wireless Internet Service Engineering (WISE) project. The project produces integrated methods and components (COTS and open source) to engineer ser- vices on the wireless Internet. The production of methods and components is driven by the development of pilot ser- vices. The methods already produced include a reference ar- chitecture, a reference development process model, as well as guidelines for handling heterogeneous mobile devices. The components include a service management compo- nent and an agent-based negotiation component. Three pilot services, i.e., a financial information service, a multi-player game, and a data management service, are being developed by different organizations. The data from the development processes of two of the pilot services are Fig. 1. Experience acquisition the basis of this study. F. BELLA ET AL.: BASELINING WIRELESS INTERNET SERVICE DEVELOPMENT - AN EXPERIENCE REPORT 3 At the beginning of each iteration, software development significance describes how the experience element has been processes are elicited as applied by the organizations; the validated and to which extent (e.g., validation through for- descriptive process models (in Figure 1, Software Process mal experiments, single case study, or survey). Model) are used to set up effort measurement programs (Measurement Plan). TABLE 1 EXCERPT OF A CHARACTERIZATION VECTOR During the development of the pilot services, the pilot performers collect data according to the measurement Customization Characteristic Pilot X factor plans. The data is validated and stored. At the end of the development cycle, baselines are built, i.e., the data col- Domain Application type Computation-intensive characteristics system lected are aggregated and quality models are built (Set of Baselines). Business area Mobile online entertain- ment services Development Project type Client New development Software characteristics Server New development Know-How Transport GSM/GPRS/UMTS protocol Implementation Client: J2ME Technology / Experience language Server: J2EE Scope Practice Element Role Technology provider, ser- vice developer 3.3 Results Characterization Significance Vector This subsection presents the results gathered from the first two development iterations. The discussion of results fo- cuses upon effort baselines from the development of the pilot services and major domain-specific risks observed. 3.3.1 Effort Baselines related to the Development of Quality Lesson the Pilot Services Model Learned This subsection discusses quality models concerning effort distribution. The quality models are gathered from the de- Process Product velopment of two pilot services. Model Model Case Study 1 Fig. 2. Overview of experience elements Context: Pilot service 1 provides a solution for real time stock tracking on mobile devices: the user can view real time quotes concerning a whole market or define his/her During post-mortem analysis sessions the baselines are own watch lists. The partner responsible for this develop- discussed with the involved parties, then interpretations ment is a provider of high end trading services on the and consequences for the next iteration are worked out Internet, aimed at banks and brokers. The pilot is the adap- (e.g., the possible evolution of the surrounding develop- tation of an existing Web-based information service. Criti- ment process). In order to get more insights of a qualitative cal usability issues arise due to the huge amount of data nature, lessons learned (Set of LL) are collected regularly by needed by a financial operator to perform an analysis and interviewing project participants at project meetings or by the small-sized display of mobile devices. Furthermore, phone. Many lessons were also gathered through the analy- since the Internet traffic on mobile devices is paid for by the sis and interpretation of baselines. Therefore, within the end user, based on data volume and not on connection context of the WISE project, different experience models time, and since frequent refresh of a large amount of finan- were applied (see Figure 2), which are an adaptation of ba- cial data is required, the adoption of the push technology sic principles of the Experience Factory [1] and QIP [2] ap- instead of the pull technology is an important issue, be- proaches. cause it avoids unnecessary data refreshes for the user. All kinds of software engineering experience are re- Most of the usability issues were addressed during the first garded as experience elements: process and product mod- iteration. The second iteration was mainly concerned with els, quantitative quality models (i.e., baselines), and qualita- implementing a solution based on the push technology. tive experience (such as lessons learned). For each experi- The life cycle model applied for developing the pilot ence element, the scope of its validity is described. service during each iteration is an iterative process model The scope consists of a characterization vector and the consisting of three phases: a requirements phase, a devel- significance. The characterization vector characterizes the opment / coding phase, and a testing phase. The ad-hoc environment in which the experience element is valid, i.e., process is characterized by extensive use of verbal commu- the context surrounding a given project (see Table 1). The nication within the development team, and little use of ex- 4 QUATIC’2004 PROCEEDINGS plicit documentation. Another important characteristic of time, of the stabilization of the process enacted by the de- the development process is the absence of an explicit design velopment team became visible and, therefore, a different, phase. This can be seen as a consequence of the fact that the more balanced, effort distribution can be observed. The overall system architecture and the related interfaces were greater amount of effort collected in the requirements phase known at the beginning of the project, since this was can be attributed to a change of the underlying process mainly the same client server architecture used to provide model description. During the first iteration, it was noticed the service on the traditional Internet. The client side was a that a part of the effort collected in the development phase prototype developed using the Wireless Markup Language was spent on performing some feasibility studies rather (WML); during the second iteration, the client was devel- than on implementing the prototype. The goal of the stud- oped using the Java 2 platform, Micro Edition (J2ME). In ies was to evaluate different mobile devices and WML con- both cases, the prototype and its high-level design were structs with respect to usability requirements. Therefore, documented after development. for the second iteration, it was decided to collect the effort Analysis: The analysis of the effort distribution observed related to the feasibility studies as requirements phase. during the first iteration and represented in Figure 3 shows that most of the effort (approx. 84%) was spent on the de- velopment phase, i.e., the creation of the first prototype. Only approx. 15% of the overall effort was spent on the 15.06% requirements phase. This can be explained as follows: the functional requirements were described at a high degree of abstraction, which was possible since they were derived 33.01% from the available Internet service and they were therefore well understood; the more challenging non-functional re- quirements, e.g., usability issues, were not formalized at all, since they were not understood at the beginning of the pro- ject and they were to be investigated with the WML proto- type. 0.15% 15.47% 51.93% requirements_phase coding_phase testing_phase Fig. 4. Effort distribution, pilot service 1, iteration 2 Although more effort was spent on integration testing than during the first iteration, most of the effort collected as testing phase was spent on documenting the integrated code. Due to the fact that the coding phase was underesti- mated, most of the system test was shifted to the third itera- 84.38% tion. During the second iteration, the development of pilot service 1 required about 200 man-days. Requirement Phase Development Phase Test Phase Case Study 2 Context: Pilot service 2 is concerned with the new devel- opment of a multi-player online game for mobile devices: Fig. 3. Effort distribution, pilot service 1, iteration 1 many users interact in a shared environment, i.e., a virtual labyrinth. The players can collect different items, chat, and Since more effort than planned was spent on the devel- fight against enemies and against each other. From a busi- opment phase, little effort remained to be spent on the test- ness point of view, games and entertainment could be, after ing phase. voice and SMS, the next killer application on the wireless During the first iteration, the development of pilot ser- Internet. The development is distributed between two dif- vice 1 required about 340 man-days. ferent teams / organizations: one organization is responsi- Figure 4 shows the effort distribution observed during ble for the development of the client on the mobile device the second iteration. The consequences of a more accurate and provides a multimedia-messaging stack on the termi- description of the development process and, at the same nal part; the other organization customizes the multimedia F. BELLA ET AL.: BASELINING WIRELESS INTERNET SERVICE DEVELOPMENT - AN EXPERIENCE REPORT 5 layer on the server side. was also reported that, due to organizational issues, less The organization responsible for the client side reaches effort than planned was spent on testing. As a consequence, CMM maturity level 3. An iterative life cycle model consist- an extensive system test must be performed during the ing of four phases (requirements phase, design phase, cod- third iteration. ing phase, and testing phase) was followed in this case within the context of each single iteration. The process is characterized by extensive use of verbal communication as well as of explicit formal documentation. 6.55% 11.75% 12.87% 27.74% 50.52% 15.19% 46.16% 29.22% Requirements Phase Design Phase Requirements Phase Design Phase Coding Phase Testing Phase Coding Phase Testing Phase Fig. 6. Effort distribution, pilot service 2 (client side), iteration 2 Fig. 5. Effort distribution, pilot service 2 (client side), iteration 1 During the second iteration, the development of the cli- Analysis (client side): During the first iteration, the fol- ent side of pilot service 2 required about 130 man-days. lowing effort distribution was observed (see Figure 5): It should be noted that effort estimates were provided at approx. 13% of the development effort was spent on the the beginning of each iteration. In order to obtain more ac- requirements phase, 29% on design, 46% on coding, and curate estimates for the second iteration, the effort distribu- 12% on testing. tion data from the first iteration were used together with Unexpected problems were reported in the requirements the first estimates as basis for the estimation process. Figure and the design phase: problems in determining which or- 7 shows how the new values for the new estimates were ganization should develop the server side led to unex- chosen from within a range between the data estimated pected low effort spent on the definition of the require- before the beginning of the first iteration and the data gath- ments; problems with the use of TCP/IP as transport proto- ered during the first iteration. col led to unexpected great effort in designing an alterna- tive protocol on the basis of UDP. The problematic behav- Effort Distribution ior of the TCP/IP protocol represents a good example of unexpected issues that may occur when applying common 60.00% Internet technologies within the wireless context. Effort (%) Finally, it was reported that less effort than planned was 40.00% spent on testing. 20.00% During the first iteration, the development of the client 0.00% side of pilot service 2 required about 140 man-days. Requiremen Design Coding Testing ts Phase Phase Phase Phase As depicted in Figure 6, during the second iteration, approx. 28% of the development effort was spent on the Estimate It. 1 26.31% 9.98% 36.09% 27.62% requirements phase, 15% on design, 50.5% on coding, and Effort It. 1 12.87% 29.22% 46.16% 11.75% 6.5% on testing. In this case, too, unexpected problems were Estimate It. 2 16.74% 22.36% 45.69% 15.21% reported during the requirements phase, since the organi- Effort It. 2 27.74% 15.19% 50.52% 6.55% zation in charge of developing the server side left the pro- Phases ject. On the other hand, due to a redesign of the graphic library that led to simplification of the further design, less Fig. 7. Overview of effort distributions and effort distribution estimates during the first two iterations effort than planned had to be spent on the design phase. It 6 QUATIC’2004 PROCEEDINGS Concerning the requirements phase, for example, During the first iteration, the development of the pilot approx. 26% was the estimate for the first iteration, 13% service 2 server side required about 130 man-days. was the effort actually spent on this phase during the first iteration, and 17% was estimated for the second iteration. The new estimated value is less than the estimate from the first iteration, but greater than the value actually measured. The estimation values were also chosen according to the 0.00% critical issues expected in the second iteration. 15.66% CPI Comparison between Iterations 1 and 2 32.53% 7 6 5 4 CPI 3 2 1 0 Requirem Design Coding Testing Overall ents Phase Phase Phase Effort CPI It. 1 5.4375 0.9082569 2.0793804 6.2509506 2.6595174 CPI It. 2 0.9373882 2.2875817 1.4047151 3.6060606 1.5533499 51.82% Fig. 8. Comparison of the cost performance indices from the first two development iterations of pilot service 2 (client side) Requirement Phase Design Phase Coding Phase Integration Phase The comparison of the Cost Performance Indices (CPI = Acceptance Test Phase planned effort / actual effort [8]) computed during the two iterations and represented by Figure 8 shows that the effort Fig. 9. Effort distribution, pilot service 2 (server side), iteration 1 estimates for the second iteration were more accurate than the estimates for the first iteration (according to the defini- During the second iteration, the organization involved in tion of CPI, an estimate is very accurate for CPI values close the development of the server side tried to apply an ap- to 1, like the estimate concerning the requirements phase of proach based on extreme programming. This makes it diffi- the second iteration; values greater than 1 indicate overes- cult to compare the effort data from the first and second timation, as in the case of the requirements phase during iteration of the server part. the first iteration). Furthermore, during the first iteration, much additional effort was spent on management-related activities, like con- 0.00% figuration management, project planning / tracking, and 12.73% project support. Due to this, the effort spent on these activi- ties was measured during the second iteration: it was seen that approx. 82% of the overall effort (1007.5 hours) was spent on development in the strict sense, whereas 18% (223 hours) was spent on management-related activities. Analysis (server side): As mentioned above, two differ- ent organizations were in charge of developing the server part of pilot service 2. During the second iteration, the sec- ond organization extended the system developed during the first iteration. Due to organizational issues, the requirements were managed by the organization responsible for the client side. As a consequence, both organizations in charge of the 87.27% server side spent little effort on defining the requirements. During the first iteration, an iterative life cycle model Exploration Phase Release Phase Planning Phase was adopted. As shown in Figure 9, effort was spent on design (32.5%), coding (52%), and integrating the client with the server part (15.5%). No requirements phase and no Fig. 10. Effort distribution, pilot service 2 (server side), iteration 2 acceptance test were performed. Unexpected problems were reported during the design phase, which were caused Furthermore, for various reasons, extreme programming by the TCP/IP protocol, whose latency was too high when was not followed strictly. Figure 10 shows, for example, used on GPRS. F. BELLA ET AL.: BASELINING WIRELESS INTERNET SERVICE DEVELOPMENT - AN EXPERIENCE REPORT 7 that the planning phase was not performed, since all re- creasing demand for appealing applications. quirements and their related priorities had already been R2: Java’s promise of code working on every platform is defined by the organization responsible for the client side. difficult to achieve: different levels of compliance with the Also, many difficulties were encountered in deploying the J2ME specification in the case of virtual machines imple- server developed by the first organization in the new or- mented by different device manufacturers can lead to great ganization’s environment; these facts did not allow many variations in performance and behavior of the same appli- short development cycles and related releases as foreseen cation running on different mobile devices. by the XP approach. As a consequence, the whole devel- R3: The maturity of the technologies specific to the wire- opment was performed at once in one big cycle. Moreover, less domain should be carefully considered: many quality the lack of experience with the test first technique led to aspects of mobile devices (file system, network access capa- unexpected effort and, although the technique was recog- bilities, memory, etc.) are of a much lower level than those nized to be very interesting, it had to be given up. of regular desktop systems. This has consequences in terms During the second iteration, the development of the of predictability of the quality of services and the develop- server side of pilot service 2 required about 80 man-days. ment process. R4: Technologies proven to be reliable when applied Comparative Analysis within the context of the traditional Internet may turn out In both case studies, the requirements phase was difficult to to be unreliable or perform poorly when used within the control due to the novelty of the domain and the fact that context of the wireless world. low level requirements and, particularly, usability-related R5: Testing wireless Internet services proved very chal- requirements (e.g., how to represent large tables on small lenging due to different reasons: The first reason are the displays) were often not well understood at the beginning. many usability issues (e.g., consistent interfaces, naviga- Feasibility studies introduced in the second iteration tion, access, etc.) related to the great diversity of devices proved to be a good means to make explicit and handle the available on the market. Most of the usability issues have to related uncertainty. be further researched due to the novelty of the domain. This uncertainty is one of the reasons why the effort Another reason is the development for future announced spent on the design phase was in all observed cases less devices: Device specifications are subject to change without than the effort spent on coding (max. 33% in the case of the notice and are usually unreliable. Another reason is that a development of the server side of pilot service 2 during the lot of effort has to be spent on setting a proper environ- first iteration). ment. Emulators represent one unsatisfactory but necessary The effort data from the development of the pilot ser- alternative solution. The main advantage of using emula- vices showed that all organizations spent most of the de- tors is the automation of the testing procedures whereas velopment effort on coding (46% - 84% of development unreliable behavior is their greatest disadvantage. effort). This seems to be plausible if the many open issues are considered that could be addressed only at the coding 3.3.3 Limits of the Study level. Concerning the validity of the quantitative part of the Testing proved very challenging due the great diversity study, i.e., the characterization of the effort distribution, of devices available on the market, the unreliability of de- Spearmint® EPGs played a major role in assuring consis- vice specifications, the low degree of automation of the test- tent views on the different development processes. These ing procedures on real devices, and the unreliability of the views and the GQM approach were very helpful in defin- available emulators. The effort spent until the end of the ing sound measurement programs that proved suitable to second iteration is considered by all the involved organiza- provide correct and meaningful data on a monthly basis. tions to be insufficient, with the consequence that most of The training of the developers responsible for collecting testing will be performed during the last iteration. data was challenging due to the widely distributed project environment and some personnel changes that occurred 3.3.2 Domain-specific Risks between the two iterations. During the first two iterations, qualitative experience was Concerning the comparability of the quantitative data, it collected by interviewing people involved in the develop- is not possible to directly compare either the numerical data ment of the pilot services. Due to the novelty of the domain, from the different pilots or all data from different iterations. the pilot partners had to deal with several risks that were This is due to the different surrounding processes applied unknown or at least not well understood at the beginning to develop the pilot services and the evolution of the proc- of the project. In the following, the main domain-specific esses during the whole project life cycle. Moreover, despite risks observed during the development of the services are an extensive literature search, no studies could be found presented. with a similar focus on effort baselines. R1: The first issue to be considered is the great diversity Concerning the generality of the results, the context of of target devices in terms of display size and mode (i.e., the single case studies defines the scope of validity of the resolution and number of colors), memory capacity, proces- baselines presented. Transforming the results for similar sor performance, and interaction mechanisms with the user contexts should be done with careful analysis of the exter- (i.e., keyboard, jog dial, cursor buttons, joystick, touch nal validity. screen, voice control, etc.). This heterogeneity makes it very Referring to the validity of the qualitative part of the difficult to reconcile the need for portability with the in- study, i.e., the collection of lessons learned, the roles played 8 QUATIC’2004 PROCEEDINGS within the pilots by the persons interviewed (mainly devel- number of classes is often the result of a great optimization opers) and the focus of the respective pilots influenced the effort and not necessarily evidence of a simpler module lessons reported. with less features. Additionally, the domain-specific risks presented in this study should be regarded as being of high significance, ACKNOWLEDGMENT since they were generalized from the lessons learned pro- vided by the individual organizations involved in the de- We would like to thank the WISE consortium, especially velopment of the pilots. the pilot partners, for their fruitful cooperation. We would also like to thank Sonnhild Namingha from the Fraunhofer Institute for Experimental Software Engineering (IESE) and 4 CONCLUSIONS Jussi Ronkainen from VTT Electronics for reviewing the This study aimed at providing effort baselines for managers first version of the article. and developers in order to give them a sense of the behav- ior of projects in the field of wireless Internet service engi- REFERENCES neering. Of course, it is important to mention that each pro- [1] Basili, V.R., Caldiera, G., Rombach H.D.: The Experience Factory, in Ency- ject is different, and that the context in which the pilots clopedia of Software Engineering (John J. Marciniak, Ed.), John Wiley & were developed must be taken into consideration before Sons, Inc., Vol. 1, pp. 469-476 (1994). making any type of analogies. [2] Basili, V.R, Quantitative Evaluation of Software Engineering The effort data from the development of the pilot ser- Methodology, in Proceedings of the First Pan-Pacific Computer vices showed that all organizations spent most of the de- Conference, Melbourne, Australia (1985). velopment effort on coding. As expected, the requirements [3] Becker-Kornstaedt, U., Boggio, D., Muench, J., Ocampo, A., Pal- phase was characterized by a great degree of uncertainty ladino, G.: Empirically Driven Design of Software Development concerning performance and availability of related tech- Processes for Wireless Internet Services. Proceedings of the nologies as well as many usability issues related to the Fourth International Conference on Product-Focused Software great heterogeneity of the devices on the market. Processes Improvement (PROFES) (2002). Testing proved very challenging due the great diversity [4] [4] Becker-Kornstaedt, U., Hamann, D., Kempkens, R., Rösch, of devices available on the market, the unreliability of de- P., Verlage, M., Webby, R., Zettel, J.: Support for the Process En- vice specifications, the low degree of automation of the test- gineer: The Spearmint Approach to Software Process Definition ing procedures on real devices, and the unreliability of the and Process Guidance. Proceedings of the Eleventh Conference available emulators. As a consequence, defect characteriza- on Advanced Information Systems Engineering (CAISE '99), pp. tion is a difficult task, and a great amount of the effort 119-133. Lecture Notes in Computer Science, Springer-Verlag. planned for the third iteration will be spent on it. It is still Berlin Heidelberg New York (1999). unclear how to characterize defects concerning usability [5] [5] Briand, L.C., Differding, C., Rombach, H.D: Practical Guide- issues. For this purpose, usability reports will be intro- lines for Measurement-Based Process Improvement. Software duced in the next iteration. Process Improvement and Practice 2, No.4, pages 253-280 (1996). The descriptive process modeling approach supported [6] Münch, J.: Muster-basierte Erstellung von Software- by the Spearmint® environment played a key role in stabi- Projektplänen, PhD Theses in Experimental Software Enginee- lizing the processes, eliciting accurate process models, and ring, Vol. 10, ISBN: 3-8167-6207-7, Fraunhofer IRB Verlag (2002). disseminating process information to the process perform- [7] Kerth, N.L.: Project Retrospectives: A Handbook for Team Re- ers. These are all necessary preconditions for meaningful views. Dorset House Publishing, ISBN: 0-932633-44-7, New York effort tracking and planning. (2001). As expected, and in spite of the accurate process models, [8] Humphrey, W. S.: A Discipline for Software Engineering (SEI effort estimation proved to be a challenging process at the Series in Software Engineering). Carnegie Mellon University, beginning. During the first iteration, the organizations in- ISBN: 0-201-54610-8. Addison-Wesley Publishing Company volved were not able to deliver effort estimates or the esti- (1995). mates they delivered turned out to be inaccurate at the end [9] Ocampo, A.; Boggio, D.; Münch, J.; Palladino, G.: Toward a Refer- of the iteration. On the other hand, effort tracking per- ence Process for Developing Wireless Internet Services. In: IEEE formed during the first iteration together with estimation Transactions on Software Engineering 29 (2003), 12, 1122-1134 : processes based on the effort data collected provided more Ill., Lit. accurate effort estimates for the second iteration. [10] Jedlitschka, A.; Nick, M.: Software Engineering Knowledge Re- How to characterize complexity and/or size of system is positories. In: Conradi, Reidar (Ed.) u.a.: Empirical Methods and still an open issue. Metrics for complexity/size can be use- Studies in Software Engineering : Experiences from ESERNET. ful for deriving effort estimation. Also, defect density Berlin : Springer-Verlag, 2003, 55-80: Ill., Lit. (Lecture Notes in measures can be built on them for controlling the testing Computer Science 2765). process. In any case, in order to estimate effort on the basis [11] Cockburn, A.: Agile Software Development: Addison-Wesley of estimates of system size or complexity, much more re- Pub. Co; ISBN: 0201699699; 1st edition (2001). search should be done. For example, regarding a metric like [12] Beck, K.: Extreme Programming Explained: Embrace Change. the number of lines of code (LOC), in the case of code writ- Addison Wesley (2000). ten for mobile devices, it was seen that the number should [13] Boehm, B.W.: A Spiral Model for Software Development and be reduced in order to improve performance; also, a low Enhancement, IEEE Computer, vol 21, No 5, pp. 61-72 (1988). F. BELLA ET AL.: BASELINING WIRELESS INTERNET SERVICE DEVELOPMENT - AN EXPERIENCE REPORT 9 [14] Buchanan, G., Farrant, S., Jones, M., Thimbleby, H., Marsden, G., Pazzani, M.J.: Improving Mobile Internet Usability. In Proceed- ings World Wide Web 10, pp. 673-680 (2001). [15] Solingen, R. van; Berghout, E.: The Goal/ Question/ Metric Method. A Practical Guide for Quality Improvement of Software Development. London, McGraw-Hill, 1999. F. Bella received his MS in Computer Science from the Technical University of Kaiserslautern, Germany, in 2002. MS Thesis: “Design and Implementation of a Similarity Analysis between Process Models in Spearmint/EPG”. He developed his MS thesis at the Fraunhofer Institute for Experimental Software Engineering (IESE) in Kaiserslau- tern. Since October 2002, Fabio Bella has been a research scientist at IESE in the department of Quality and Process Engineering. Since June 2003, he is a Provisional SPICE Assessor. Bella’s research inter- ests in software engineering include: (1) modeling and measurement of software processes and resulting products, (2) software quality assur- ance and control, (3) technology transfer methods, and (4) software process assessments. J. Münch received his PhD degree (Dr. rer. nat.) in Computer Science from the University of Kaiserslautern, Germany. Dr. Münch is Depart- ment Head and Competence Manager for Quality and Process Engi- neering at the Fraunhofer Institute for Experimental Software Engineer- ing (IESE), Kaiserslautern. Since November 2001, Dr. Münch has been an executive board member of the temporary research institute SFB 501 "Development of Large Systems with Generic Methods" funded by the German Research Foundation (DFG). Dr. Münch’s research inter- ests in software engineering include: (1) modeling and measurement of software processes and resulting products, (2) software quality assur- ance and control, (3) technology evaluation through experimental means and simulation, (4) generic methods for the development of large systems, (5) technology transfer methods. He has been teaching and training in both university and industry environments, and also has significant R&D project management experience. Jürgen Münch is a member of the IEEE Computer Society and the German Computer Society (GI). A. Ocampo Alexis Ocampo received his MSc degree in Computer Science from Los Andes University, Colombia, in 1999 and his title as Systems Engineer from Industrial University of Santander, Colombia, in 1997. Since February 2002, Alexis Ocampo has been a research sci- entist at the Fraunhofer Institute for Experimental Software Engineering (IESE), Kaiserslautern, in the department of Quality and Process Engi- neering. Before that, he worked for 5 years as a research-developer on new technologies and methodologies with the software company Hein- sohn Associates, Bogota, Colombia. His master thesis entitled “Imple- mentation of PSP in the Colombian Industry: A case study” was devel- oped within this company. He also worked as an instructor at the Uni- versity of Los Andes in the Department of Systems and Computation. Alexis Ocampo’s research interests in software engineering include: (1) modeling and measurement of software processes and resulting prod- ucts, (2) software quality assurance and control, (3) technology transfer methods.