=Paper=
{{Paper
|id=Vol-3272/paper3
|storemode=property
|title=A Survey on COSMIC Students Estimation Challenge
|pdfUrl=https://ceur-ws.org/Vol-3272/IWSM-MENSURA22_paper3.pdf
|volume=Vol-3272
|authors=Tuna Hacaloglu,Bilge Say,Huseyin Unlu,Neslihan Kucukates Omural,Onur Demirors
|dblpUrl=https://dblp.org/rec/conf/iwsm/HacalogluSUOD22
}}
==A Survey on COSMIC Students Estimation Challenge==
https://ceur-ws.org/Vol-3272/IWSM-MENSURA22_paper3.pdf
A Survey on COSMIC Students Estimation Challenge
Tuna Hacaloglu1 , Bilge Say1 1 , Huseyin Unlu2 , Neslihan Kucukates Omural3 , Onur Demirors2
1
Atilim University, Kızılcaşar, İncek, Gölbaşı, Ankara, 06830, Türkiye
2
Izmir Institute of Technology, Gülbahçe, Urla, İzmir, 35430, Türkiye
3
AYESAŞ, Üniversiteler Mahallesi, Çankaya, Ankara, 06800, Türkiye
Abstract
Software project management is a significant software engineering practice that is highly
related to achieving software-specific project goals. This study aims to share students’
perceptions of incorporating an international software estimation challenge called “COSMIC
Students’ Estimation Challenge” into a software project management course. For this aim,
students were taught the COSMIC Functional Size Measurement method and entered the
competition. After the competition, a questionnaire asking for the students’ opinions was
collected. The objective of the research is to get an insight into to what extent incorporating
this type of competition activity -a challenge- can contribute to students’ learning perceptions.
In the long run, the findings can contribute to creating a foresight about making the necessary
curriculum arrangements to form a more up-to-date and dynamic education plan by including
the methods applied in the software industry in Software Engineering education. The results
suggest that this kind of competition experience and preparation is helpful for students to learn
the COSMIC method.
Keywords 1
project management education, software engineering education, COSMIC, challenge,
gamification
1. Introduction
Software project management is one of the drivers of developing high-quality software products on
time and on budget. Software projects often exceed the anticipated budget and generally result in time
delays. To raise awareness of the importance of this concept, nowadays, many software engineering
departments worldwide have already incorporated a standalone software project management course
into their curriculum. In addition, several recent studies have been found in software engineering
literature on teaching software measurement processes by exploring different ways, such as using
gamification [1, 2].
In this respect, software size stands out as an important input to consistently making project related
estimations. In addition to various methods for size measurement, there is a method called COSMIC
Functional Measurement (FSM), which ISO also recognizes as a standard [3]. COSMIC Consortium
organizes various international activities for the dissemination of the method. One of these activities is
the COSMIC Student Challenge 2022, which was held for the third time this year on April 9th, 2022.
Students participated in this competition in teams and applied the steps of finding the size of a software
project using the COSMIC FSM method. Utilizing the approximation techniques suggested by the
method, if the requirements do not contain sufficient details, they estimated the effort using the size that
they have measured.
Within the scope of the “Software Project Management and Economics” course in the Software
Engineering Department of a university in Turkey, the COSMIC FSM method was presented to students
as a part of the project estimation topic. In addition to the materials described in the course content,
Proceedings Name, Month XX–XX, YYYY, City, Country
EMAIL: email1@mail.com (A. 1); email2@mail.com (A. 2); email3@mail.com (A. 3)
ORCID: XXXX-XXXX-XXXX-XXXX (A. 1); XXXX-XXXX-XXXX-XXXX (A. 2); XXXX-XXXX-XXXX-XXXX (A. 3)
© 2020 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Wor
Pr
ks
hop
oceedi
ngs
ht
I
tp:
//
ceur
-
SSN1613-
ws
.or
0073
g
CEUR Workshop Proceedings (CEUR-WS.org)
presentations and videos prepared by the COSMIC Consortium were also offered to the students. The
students then participated in the “COSMIC Students’ Estimation Challenge” and tried themselves in
the competition.
After the challenge, we organized a survey to investigate the students’ feedback regarding the
contribution of the education given during the lectures and the experience after an international
measurement competition. In this article, we present the results of the survey in several dimensions,
such as prior knowledge about the method, preparation for the challenge, the challenge experience,
team scores, and suggestion for improvement suggestions from the students’ perspectives.
The remainder of this paper is organized as follows. Section 2 reviews related research regarding
software project management and software engineering education. In Section 3, our research
methodology for the survey is explained. In Section 4, the results of the survey are presented. Section
5 discusses the survey findings. The main conclusions, limitations, and future work are presented in
Section 6.
2. Background
In the literature, various studies are proposed to enhance software engineering (SE) education. Based
on a recent study presented in [7,] engaging students is among the main challenges regarding software
engineering education, and designing practical activities is also mentioned as one of the difficult tasks.
To improve the teaching activities in SE, several approaches are adopted, including gamification
techniques, problem-based learning, role-playing, flipped classroom, serious game, etc. [7].
A handful of studies on improving teaching software size measurement were presented in the
literature. For example, in [1], the authors proposed a serious game for students to practice the COSMIC
FSM measurement. Based on their study, the authors found that the most of the participant students
think that the game proposed helped them learn how to use COSMIC for estimation.
Later authors in [2] presented another study in which they presented a proposal using gamification
with a serious game to instruct the software measurement process. The authors concluded that this type
of methods were beneficial for teaching software measurement due to gamification’s competitive and
collaborative characteristics.
One of the main properties of a software in terms of measurement is its size. Software size can be
described in three core attributes: length, complexity, and functionality [4]. Measuring size in terms of
functionality has many benefits over using the other attributes, where early estimation is one of them.
Functional size measurement-related studies originate from Albrecht’s Function points in 1979. Many
other methods, such as IFPUG, Mark II, NESMA have been published afterwards [5].
COSMIC FSM method was officially introduced in 2001 as a second generation FSM method which
attracted attention by being applicable to a wide range of software applications such as business
applications, service-oriented systems, real-time systems, etc., and by being independent of tools and
technologies [6]. COSMIC FSM method is based on counting data movements. The data movements
are described as Entry (E), Exit (X), Read (R), and Write (W). The method suggests that a functional
user requirement comprises one to many functional processes. These functional processes include a set
of data movements. The size of a functional process is the sum of its data movements. Accordingly, the
size of a given functional requirement is the sum of the size of its functional process. Consequently, a
software system’s functional size is the sum of the size of its functional requirements [3].
The method is perfectly applicable when functional requirements contain required details. In
addition, the COSMIC FSM method also has some approximation techniques when the given
requirement set does not contain sufficient details in the requirement set. It has also a guideline for
measuring non-functional requirements.
The COSMIC Consortium provides a set of manuals and materials, including case studies, to instruct
the measurement method. All these materials can be accessed for free on the WEB page of COSMIC2.
2
https://cosmic-sizing.org/
3. Research Methodology
In this study, the survey is used as the research methodology. The main goal of this survey is to
explore the contribution of this type of competition to the students’ perception about how much they
have learned software size measurement and estimation, and how much this competition raised their
awareness of this topic. We accordingly derived the following research questions:
• RQ1. Does COSMIC Students’ Challenge contribute to learning size measurement and estimation
concept?
• RQ2. Does COSMIC Students’ Challenge raise awareness about the utilization of size
measurement in software project management?
As for our sampling method, we chose third-year software engineering students taking the “Software
Project Management and Economics” course at a university in Turkey. Other student groups could be
used, but since software estimation is one of the topics of software project management course, this
student sample was the most appropriate one to teach the COSMIC FSM method.
Considering the number of questions, we decided to divide the survey into sections to make it clear
and understandable. The survey consisted of four main sections:
• Participant Information
• Prior Knowledge
• Preparation for the Challenge
• COSMIC Challenge Experience
The survey was offered to students after the challenge. It was offered online and using the Google
Forms survey platform. Approval from the Institutional Review Board was also taken for applying the
survey. Participation of the survey was voluntary but to engage the student interest, two points were
offered as a bonus point for the overall course grade.
4. Results
In the challenge, we had 22 participating teams, including a total of 65 students with a team size of
3. Participation in the survey was voluntary. Accordingly, 56 students from 22 groups participated in
our survey. The survey comprises six sections: content approval, participant information, prior
knowledge, preparation for the challenge, challenge experience, and additional comments. The results
of each section are summarized in the following sections.
4.1. Prior Knowledge
We first asked about the participants’ prior knowledge of the COSMIC method based on Bloom’s
taxonomy (see Figure 1). 5.4% of the participants have applied, 10.7% know, and 19.6% have
previously heard of the COSMIC method. The majority (64.3%) of the participants have no prior
knowledge about the COSMIC method.
Figure 1: Prior knowledge of the COSMIC method
In the second question, we asked a more generic question about the prior knowledge of software
size measurement. The results show that the prior knowledge of software size measurement is more
than the COSMIC method (see Fig. 2). 8.9% of the participants have applied, 17.9% know, and 28.6%
have no prior knowledge where 44.6% of the participants have previously heard of the COSMIC
software size measurement method.
Figure 2: Prior knowledge of the software size measurement
4.2. Preparation for the Challenge
The first question asks whether the participant has attended the challenge preparation lectures or not
(see Figure 3). Similar to our observation from the lecture, most students (91.1%) participated in the
lecture, while 8.9% did not attend.
Figure 3: Attendance of the COSMIC Challenge Preparation Lecture
Secondly, we asked how they prepared for the challenge (see Figure 4). 62.5% of the participants
read given COSMIC manuals, 62.5% read challenge slides provided by COSMIC, 50% measured
sample projects, and 42.9% watched challenge videos available online provided by COSMIC.
Figure 4: Preparation for the challenge
Thirdly, we asked to explore the time spent preparing for the challenge (see Figure 5). 50% spent 1-
3 hours, 30.4% spent 4-6 hours, and 19.6% spent less than one hour. None of the students spent more
than 6 hours preparing for the challenge.
Figure 5: Time spent preparing for the challenge
Lastly, we asked about the satisfaction level of the students’ preparation for the challenge (see Figure
6). 35.7% of the participants are neutrally satisfied, 30.4% are satisfied, 17.9% are unsatisfied, 10.7%
are very satisfied, and 5.4% are very unsatisfied.
Figure 6: Participants' preparation satisfaction
4.3. COSMIC Challenge Experience
In this section, we asked questions about their experience with the challenge. Firstly, we asked them
to rate the difficulty of the challenge (see Figure 7). 33.9% of the participants voted neutral, 32.1%
found the challenge difficult, and 26.8% found it very difficult. A small group of participants found the
challenge as easy.
Figure 7: The difficulty of the challenge (1: very easy; 5: very difficult)
Secondly, we asked them to rate the given time for the challenge (see Figure 8). 37.5% found it
insufficient, 7.1% found it very insufficient, while 17.9% found it sufficient, and 5.4% found it very
sufficient. The percentage of neutral participants is 32.1%.
Figure 8: The sufficiency of the given time in the Challenge (1: very insufficient; 5: very sufficient)
Thirdly, we asked which part of the challenge was the most difficult (see Figure 9). Although there
is no weight on a specific part, the results are as follows: sizing the detailed functional requirements
(28.6%), sizing the non-functional requirements (25%), estimating the effort to develop both functional
and non-functional requirements (19.6%), developing an estimation model from the historical data
(19.6%) and approximating the size of the requirements (12.5%).
Figure 9: The most difficult part of the challenge
Fourthly, we asked whether the challenge helped them learn the COSMIC method or not (see Figure
10). The results show that most participants found the challenge helpful (41.1%) or extremely helpful
(25%). Accordingly, 25% voted neutral, and 8.9% of the participants found it not helpful.
Figure 10: The helpfulness of the challenge (1: not at all helpful; 5: extremely helpful)
Fifthly, we asked the participants’ opinions on the usefulness of the COSMIC method in real-life
projects (see Fig. 11). 28.6% found it very useful, 25% found it useful, while 28.6% voted neutral.
Figure 11: The usefulness of the COSMIC method (1: strongly disagree; 5: strongly agree)
Lastly, we asked participants whether the challenge has changed their perception of the usability of
the COSMIC method in real-life software projects (see Fig. 12). 35.7% of the participants agree, 28.6%
are neutral, 19.6% strongly agree, 8.9% disagree, and 7.1% strongly disagree.
Figure 12: Perception of the usability of the COSMIC method (1: strongly disagree; 5: strongly
agree)
4.4. Team Scores
As we mentioned, 22 teams participated in the challenge; however, one team did not submit their
work. Thus, 21 teams remained. The statistical summary of the team scores is presented in Table 1.
Table 1
The statistical summary of team scores
Number Avg. Max. Min. Standard
of score score score deviation
teams
21 42.24 84 9 18.23
5. Discussions
We will discuss our findings based on participants’ comments and suggestions in three sections:
recommendations for improvement of the challenge, suggestions for improvement of the study material,
and additional comments of the participants.
5.1. Suggestions for Improvement of the Challenge
We asked an open-ended question to the participants about their suggestions for improvement of the
challenge. The participant suggestions are listed as follows:
• In parallel to survey results, many participants found the duration of the challenge not sufficient.
They expressed that the duration could be longer as there are many tasks.
• Another common suggestion was about the number of tasks in the challenge. As the time was
insufficient for the participants, they suggested that the number of tasks could be less.
• Another suggestion was about team size. The participants suggested that all teams should have a
fixed size for equality.
• Some participants found preparing a presentation quite unnecessary. They suggested submitting a
report instead of a presentation which suits the answers better.
• Participants also suggested that there should be a better explanation for the tasks. Similarly, some
participants found the introduction presentation at the beginning of the competition not sufficient. They
suggested that the challenge should be explained in more detail.
• The last suggestion was about the date of the challenge. They suggested locating the challenge at
the end of the semester as they can have more time to focus on the challenge.
5.2. Suggestions for Improvement of Study Materials
Similarly, we also asked an open-ended question to the participants about their suggestions for
improvement of the study materials. The participant suggestions are listed as follows:
• A common comment about the study materials was about including concrete examples in the
materials. They suggested including examples similar to the challenge in study materials. They found
the challenge much more complex than the examples in the study materials.
• Another suggestion was about the release date of the study materials. They suggested that study
materials should be released earlier to provide them more time to be prepared.
In order to improve software project management courses in software engineering curriculums, we
infer that the syllabus can be updated in accordance with the challenge components such as sizing
functional requirements, making the approximation, sizing non-functional requirements, and
performing the estimation.
5.3. Additional Comments
The additional comments of the participants show that COSMIC Challenge reached its goal. Some
other comments of the participants are given as follows:
• “Thank you for giving us this chance. If I specialize in project management in the future, I'll be
able to say, "I've joined COSMIC.”
• “It is beneficial to understand estimating project size.”
• “This was an extraordinary experience for me. It was an instructive and entertaining activity and
reinforcing work; thank you very much.”
• “For the students who will participate in the challenge, I strongly recommend that you do not solve
each step together since the duration is really limited, do each part separately and decide the work
assignment according to the performance of your team members.”
• “It was fun working together with friends.”
• “It was a fun event.”
• “It was a different and unique experience.”
• “Thanks to our professors, we became aware of this COSMIC challenge and participated in the
challenge. It was beneficial for us. I think we will use these things we learned in our future business
life, and I would like to thank our professors.”
• “It was both a fun and instructive challenge. I am happy to participate and experience.”
• “I like the challenge; I hope I will join the challenge again.”
• “It was a decent challenge.”
• “The challenge was perfect; it added a lot to me, especially about project management.”
6. Conclusion
This year, COSMIC Consortium organized the third Students Estimation Challenge on April 9th,
2022. Apart from the last two challenges, the consortium provided study materials (slides and videos)
for the participants this year. 53 teams participated in the challenge, including 234 students from
Turkey, Canada, Egypt, Cameroun, and Mexico.
We supervised 22 teams from Turkey. After the challenge, we organized a survey to explore the
feedback of participants from Turkey about the challenge. One of our supervised teams won the second
prize. In this paper, we presented the survey results and findings.
Considering RQ1, we can say that COSMIC Challenge reached its goal. Participants commented
that they improved their knowledge and skills in software estimation. The incorporation of this practical
experience to a software project management course made the subject more understandable and excited
the attention of the students. Regarding RQ2, the results suggest that the challenge increased the
awareness of the students about size measurement and estimation. Moreover, the findings suggest that
study materials can be improved by incorporating concrete examples. A potential future study can
include statistical analyses such as correlation analysis to investigate the relationship between the
students’ preparation, students’ opinions, and the scores they get.
With this article, we aimed to gather the students’ opinions regarding the challenge. COSMIC
Consortium improves the challenge each year. We believe that the results of this survey will be an
essential input and can give insights to improve the challenge next year.
7. Acknowledgment
We would like to offer our special thanks Prof. Dr. Alain Abran and COSMIC Committee for their
valuable contribution for the accomplishment of this challenge and for the preparation of training
materials.
8. References
[1] Marín, B., Vera, M., & Giachetti, G. (2019). An Adventure Serious Game for Teaching Effort
Estimation in Software Engineering. In IWSM-Mensura (pp. 71-86).
[2] Furtado, L. S., & Oliveira, S. R. B. (2020, October). A teaching proposal for the software
measurement process using gamification: an experimental study. In 2020 IEEE Frontiers in
Education Conference (FIE) (pp. 1-8). IEEE.
[3] COSMIC Measurement Manual Version 4.0.2. The Common Software Measurement International
Consortium (2017).
[4] N. E. Fenton and S. L. Pfleeger, “Software metrics: a rigorous and practical approach PWS
Publishing Co,” Boston, MA, USA, 1997.
[5] C. Gencel and O. Demirors, “Functional size measurement revisited,” ACM Trans. Softw. Eng.
Methodol., vol. 17, no. 3, pp. 1–36, 2008.
[6] Abran, A., Symons, C., Ebert, C., Vogelezang, F., & Soubra, H. (2016). Measurement of software
size: Contributions of cosmic to estimation improvements. In The International Training
Symposium, At Marriott Bristol, United Kingdom (pp. 259- 267).
[7] Ouhbi, S., & Pombo, N. (2020, April). Software engineering education: Challenges and
perspectives. In 2020 IEEE Global Engineering Education Conference (EDUCON) (pp. 202-209).
IEEE.
[8] Alhammad, M. M., & Moreno, A. M. (2018). Gamification in software engineering education: A
systematic mapping. Journal of Systems and Software, 141, 131-150.