<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Classifying the Measures of Software Sustainability</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Shola Oyedeji</string-name>
          <email>shola.oyedeji@lut.fi</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Ahmed Seffah</string-name>
          <email>ahmed.seffah@lut.fi</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Birgit Penzenstadler</string-name>
          <email>birgit.penzenstadler@csulb.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Computer Engineering, and Computer Science California State, University Long Beach (CSULB) Long</institution>
          ,
          <addr-line>Beach, California</addr-line>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>LUT School of Engineering (LENS), Lappeenranta University of Technology</institution>
          ,
          <addr-line>Lappeenranta</addr-line>
          ,
          <country country="FI">Finland</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>- Energy efficiency is one of the very few measures widely used for evaluating green and sustainable software systems. This paper investigates the current measures of software sustainability from the four different software sustainability perceptions: Sustainability in Software Development, Green Software Systems, Software for Sustainability, Sustainability of the Software Eco System and Software Sustainability Dimensions (Economic, Social, Individual, Technical and Environment). While exploring the literature on green and sustainable software systems, measures of green software and software sustainability were identified, compiled and classified according to the four sustainability perceptions.</p>
      </abstract>
      <kwd-group>
        <kwd>green software</kwd>
        <kwd>sustainable software</kwd>
        <kwd>measures</kwd>
        <kwd>sustainability</kwd>
        <kwd>sustainability perceptions</kwd>
        <kwd>green measures</kwd>
        <kwd>software measurement</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>I. INTRODUCTION</title>
      <p>
        Sustainability is now one of the world major challenge
[
        <xref ref-type="bibr" rid="ref11">1</xref>
        ][2]. The United Nations Sustainable development Goals
(SDGs) shows the importance of sustainability in all facet of
human lives and development. Today’s economy rely on
information and communications technology (ICT) in which
software is a key factor and catalyst for all economic
activities and a major driver linking all sectors. As stated in
an Ericsson report that ICT can help reduce the global
greenhouse gas (GHG) emissions by 15% [3]. Currently ICT
itself contributes an estimated 2% to the global CO2
emissions and accountable for approximately 8% of the
European Union (EU) electricity consumption [4]. This
shows ICT can a huge potential to help support sustainability
and Green [5] but at same time it is important to explore
avenues to make ICT domain more green and sustainable
because of its huge impact on sustainability. Finding ways to
properly evaluate software in regards to green and
sustainability will provide avenues to reduce the current
negative impacts of ICT.
      </p>
      <p>This research explores the ongoing perceptions in the
software engineering domain with the goal to identify the
current and future measures used in the evaluation of green
and sustainable software. Via triangulation of data from
diverse sources, the measures are clustered into the four
perceptions of sustainability in software engineering and
sustainability dimensions. The long term goal of this research
is to answer the following challenging questions: what are
the current measures used in evaluating green and
sustainability aspects of software systems and how can these
measures be grounded in the software sustainability
measurement theory.</p>
      <p>Furthermore a checklist and guide approach that
demonstrates how to include the objective of environmental
sustainability from the very early steps of software
development can assist in identifying key stakeholders. This
will facilitate the ability to accommodate new objectives of
improving the environmental sustainability of software
systems [13]. Roher et al. [14] suggests the use of
sustainability requirement patterns (SRPs), which will
provide software engineers with guidance on how to write
specific types of sustainability requirements with the goal to
overcome the barriers of incorporating environmental
sustainability into the requirements engineering process.</p>
      <sec id="sec-1-1">
        <title>B. Green software system</title>
        <p>Green software is an environmentally friendly software that
consumes less energy, provides less impacts on environment
and support carbon management [15]. It is also software that
fulfils high level requirements, ensuring the software
engineering process, maintenance, and disposal saves and/or
reduces resource waste [16] [17]. Green software is divided
into four parts: software that is energy efficient during
execution, software that are embedded to execute and
support smart operations in green manner, software to
produce environment viable products and policies [18]. The
goal of green software engineering is to provide supports for
efficient consumption of natural resources while
continuously monitoring, evaluating and optimizing the
aftermath effects caused during the software system life
cycle [19].</p>
        <p>Erdélyi [20] paper provides an overview of different
activities and advice on what to do in order to develop green
software which uses energy efficiently and produce less
waste. The paper highlights three ways software engineering
can be green such as: produce green software, produce
software to support environmentally consciousness (green by
software) and produce less waste during development.</p>
        <p>Dick et al. [21] provides basis for the right way to
engineer green software systems using development process
that ensures that the positive and negative effects of the
software is continuously monitored and evaluated in order to
optimize the software over its life cycle to be more green
(environmental friendly).</p>
        <p>Colmant et al. [22] presented researches on to improve
the software-energy efficiency on multi-core systems.
Colmant et al. [28] motivations were driven by the huge
impact of the ICT on the world CO2 emissions which
represents 2%. Calero et al. [4] highlights some of the
meanings of green software notably a software that
consumes less energy to run and produces as little waste as
possible during its development and operation. Largely,
research on green software has focused more on energy
consumption and environmentally friendly software systems.</p>
      </sec>
      <sec id="sec-1-2">
        <title>C. Software for Sustainability</title>
        <p>There has been some interest in various domains such as
manufacturing, energy sector, transportation and for different
application in recycling, product packaging, data center
setup, gas emissions. Some of the good examples are in grid
computing, in Human Computer Interaction (HCI) to change
the habit of people.</p>
        <p>In [23], authors presented a software system that support
sustainable lifestyles with an example of a domestic plant
guild to show how sustainable human systems can
effectively support a sustainable lifestyle, which can reduce
the cost of living as well as the ecological footprint.
Penzenstadler et al. [24] highlights vision for systems that
will be supporting sustainability in the future (2029) with a
set of fictional abstracts around the concepts of
sustainability, complexity, collapse, and resilience of ICT
systems.</p>
        <p>Software can also provide support for sustainability in
different domains such as:
x The use of software systems for tracking gas
emissions
x Software for climate and disaster prediction
x Smart infrastructural management software
x Enterprise carbon and energy management software
x Smart transportation software to reduce CO2
emissions.</p>
        <p>Sustainability Knowledge and Learning
Management software
Software for environmental awareness on wildlife
and plants</p>
      </sec>
      <sec id="sec-1-3">
        <title>D. Sustainability of the software ecosystem</title>
        <p>Today software systems are the pillars of the economy,
the software eco system is probably the biggest system in the
world we human created. Software eco system has been
defined according to Jansen et al. [25] as a set of actors
functioning as a unit and interacting with a shared market for
software and services, together with the relationships among
them. These relationships are frequently underpinned by a
common technological platform or market and they operate
through the exchange of information, resources and artifacts.</p>
        <p>Thus, the sustainability of software ecosystem involves
the sustainment of the global system of software systems and
services covering aspect of how different sub systems form a
huge interconnected system and all the interactions. It covers
all different components such as hardware, software and
network that is used to resolve complex relationships among
companies/organizations in all the different sectors and
industries [26].</p>
        <p>Sustainability of the software ecosystem entails how can
the system of software systems endure with the evolving user
requirements and usage overtime with less negative impact
on the environments, social, technical and humans. This
means the ability of software ecosystem to continue to
function and evolve irrespective of any glitch is some part of
the ecosystem and should continuously fulfil users’ needs.</p>
        <p>III. PERCEPTIONS OF SUSTAINABILITY IN/FOR SOFTWARE</p>
        <p>SYSTEMS</p>
        <p>In this research, we defined sustainability as a quality
construct in the same ways other factors are defined (see, for
example, the ISO 25 000 family of standards). In our
perception sustainability aims to create balance in the way
humans live, produce, and use products and services
(resources) with the objective to have less negative impact on
the environment and promote the wellbeing of all living
species. This means the capacity of software systems to
endure in certain ecosystems under current and future
conditions while satisfying the needs of users today and
tomorrow with minimum negative impact on the
environment; at the same time supporting business growth
and societal values.</p>
        <p>Currently, the dimensions of software sustainability are
known and classified into five: economic, environment,
social, individual and technical [27] but there is currently no
clear categorisation for the perceptions of sustainability
in/for software engineering. This section explains the
categorization of software sustainability perceptions based
on the literature review from the background section.
Software sustainability evolution today can be perceive from
one of the following perception (see Figure 1); Sustainability
in Software Development, Software for Sustainability, Green
Software Systems, Sustainability of Software Ecosystems.
x</p>
        <p>Sustainability in software development
(Development): this refers to the processes
involve in the development of software
(software development life cycle).
x
x
x</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>Software for sustainability (Usage): how software are used to support sustainability, an example is a software in fridge to minimize energy wastage (embedded software).</title>
    </sec>
    <sec id="sec-3">
      <title>Green software systems (Focused impact): software systems that uses less energy resource and promotes policies that supports green awareness.</title>
    </sec>
    <sec id="sec-4">
      <title>Sustainability of software ecosystems (Net</title>
      <p>effect): This is the total impact of the entire
software ecosystem (systems of system)</p>
      <p>The advancement of software sustainability from the four
perceptions has received different level of research attention
and contributions. Sustainability in software development,
Green Software system we observed has the most important
advancement in research compared to software for
sustainability and sustainability of software ecosystem that
were not fully explored.</p>
      <p>Figure 1 portrays the categorization of software
sustainability perceptions.</p>
    </sec>
    <sec id="sec-5">
      <title>IV. MEASURES AND MEASUREMENT OF GREEN AND</title>
      <p>SUSTAINABILITY IN/FOR SOFTWARE SYSTEMS</p>
      <p>This section presents different research work relating to
green and sustainable software system measures and
measurement. According to Britannica [28], measurement is
the science of assigning of a quantity, either quantitative or
qualitative, to a characteristic of an object or event, while
making it comparable to other objects or events. Here object
is the software and event is the development process.</p>
      <p>Sustainability measurement is still a new idea [29] [30] [31]
[32]. Indeed, Lami et al. [31] stated that there are few studies
about ‘what’ aspects of sustainability to measure and ‘how’
to do it. Calero et al. [33] highlighted that nowadays,
sustainability is a key factor that should be considered in the
software quality models, though there has less research
channelled towards sustainability measurement. Seacord et
al. [29] stated that planning and management of software
sustainment is impaired by a lack of consistently applied,
practical measures, and there is no central theoretical
framework on measurement of software sustainability.</p>
      <p>One of the most referenced sustainability measurement
model for software system is the GREENSOFT Model [7]. It
is a conceptual reference model for “Green and Sustainable
Software”, which has the objective to support software
developers, administrators, and software users in creating,
maintaining, and using software in a more sustainable way
[34]. Another software sustainability measurement approach
is the Sustainable Business Goal Question Metric (S-BGQM)
[35]. It encourages the incorporation and measurement of
sustainability during the entire software system development
processes. Kramer [36] also wrote about sustainability
measurement by proposing some set of questions that should
be addressed by any sustainability framework.</p>
      <p>A study for monitoring software energy hotspot proposed
power model for software energy cost formula as Esoftware
= Ecomp +Ecom +Einfra, where Ecomp is the computational
cost (i.e., CPU process- ing, memory access, I/O operations),
Ecom is the cost of exchanging data over the network, and
Einfra is the addi- tional cost incurred by the OS and runtime
platform (e.g., Java VM) [37]. The study focused on energy
consumption of CPU and network demanding software at
different levels of granularity. Also, the formula proposed for
software energy efficiency (EF) is
UsefulWorkDone/UserdEnergy [38]. This generic measure
provide a way for evaluating the energy consumption of
different software parts and modules using white box testing
to measure which parts are consuming more energy and to
see which parts can be optimized for efficient energy usage.</p>
      <p>In order to facilitate research on energy usage attribution,
software energy footprint lab was setup to provide insight on
energy footprint measurements with results interpreting
hardware dissipation profiles for various servers under
different kinds of software stress [39]. This shows the
relations between different hardware resource and the
amount of resource required by the running software in
relation to the power consumption.</p>
      <p>Furthermore, a support tool is presented to analyze
legacy systems in order to estimate the energy consumption
and detect parts of the system with higher energy
consumption. Using the profiling technique, the tool
instrument legacy Java systems in order to keep track of its
execution. This information, together with the energy
consumption, enables the engineer to analyze legacy system
consumption detecting energy peaks in the system [40].</p>
      <p>Additionally, a modular Energy-Aware Computing
Framework (EACOF) is proposed as a way to allow access
to energy consumption information of software through API
calls. The EACOF is separated into two task for collection
and utilization of dynamic energy consumption data which
reduce development and maintenance overhead required for
the successful completion of each task[41]. Another
approach is also proposed for monitoring power
consumption of software in order to assist software designers
and developer to reduce software power consumption and
have better energy efficiency [42]. This approach currently
monitors power consumption at source code level, this
approach will provide better insights on software energy
consumption if extended to the hardware running the
software.</p>
      <p>As summarized in Table 1 and the research work detailed
in [43] [44] [45] and [46], other measures of green and
sustainable software have been on software and hardware
energy consumption with less research for measures
covering software sustainability dimensions such as
individual, social, economic and software sustainability
perceptions (Software for sustainability and Software
ecosystem).</p>
      <p>The measures detailed in Table 1 are structure based on
categorization of software sustainability and green measures
for software sustainability dimensions and the four
sustainability perceptions. Each column after the main title
has a “YES or No” to indicate if the proposed measure in the
research paper cover any of the categories listed in Table 1.</p>
      <p>Most of the measures descriptions does not explicitly
indicate that the authors considered sustainability
dimensions. Base on the descriptions and explanations of the
authors for all measures, we have categorized those measures
according to the right sustainability dimension (Economic,
Social, Individual, Technical and Environment) to show how
it relates to the four sustainability perceptions.</p>
      <p>MEASURES FOR GREEN AND SUSTAINABLE SOFTWARE LINKED TO SUSTAINABILITY DIMENSIONS AND PERCEPTIONS</p>
      <p>Table 1 provides details of measures attributed to green
and sustainable software. From Table 1, it can be identified
that most measures focused on energy efficiency or power
consumptions. With most focus on green software, there is a
limitation on having a holistic approach towards software
sustainability measurement. The measures of software
sustainability should consider the following:
x Human (End users) system interaction: involves the
measures of the system sustainability based on how it
impacts on users and their level of awareness about
sustainability and green. It entails the well-being of the
software users’ community and the changing of the
human mindset.
x Software system developers: evaluate the sustainability
of the processes and practices for the development and
integration of sustainability in software systems.</p>
      <p>One of the key question/concern that should be clearly
answered by a sustainability measurement framework is the
difference between the different scales of software
measurement and the interpretation of these scales of
measurement for sustainability. The problem of software
sustainability measurement is not only in measuring but
Yes</p>
      <p>No</p>
      <p>No</p>
      <p>No
rather giving meaningful interpretation of what the
measurement means. For example today, fridges are
categorized using A+, A++ and A+++ for quantifying and
measuring its energy efficiency. Normally A+ consumes less
energy, A++ has better energy efficiency than A+ and A++
has the best energy efficiency in today market. According to
the EU Directive 92/75/EC which established an energy
consumption labelling scheme [52], there are different
descriptions of the measures that quantify why Fridge is
labelled A+, A++ or A++ based on its energy consumption.</p>
      <p>In the same line, there is need for a foundation or framework
to ground the different measures for software sustainability
measures and measurement with clear interpretation.</p>
      <p>Currently, there is not enough firm scientific basis for
important choices on how sustainability related factors
should be defined and measured, the varying purposes for
which the measures are used. This makes it difficult to
effectively and efficiently evaluate software sustainability
using the right measures.</p>
    </sec>
    <sec id="sec-6">
      <title>VI. CONCLUSION</title>
      <p>
        In this position paper, we summarized the research
results on the categorization of software sustainability
perceptions. Using the identified four perceptions of software
sustainability, we referenced the current measures to each of
the four perceptions. The major focus of all identified green
and sustainable software measures are on green software.
Energy efficiency has received the most attention. Research
work is needed to identify and assess the validity of other
measures related to the other perceptions. Research on
measures of sustainability has to be grounded in the tradition
and theory of software measurement. This requires
considering software sustainability as a quality attribute and
define it in the same way other attributes are defined.
[
        <xref ref-type="bibr" rid="ref11">1</xref>
        ]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
      </p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <given-names>United</given-names>
            <surname>Nations</surname>
          </string-name>
          ,
          <source>World Economic and Social Survey</source>
          <year>2013</year>
          .
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          <string-name>
            <surname>Qual.</surname>
          </string-name>
          , vol.
          <volume>4542</volume>
          , no.
          <source>January</source>
          , pp.
          <fpage>247</fpage>
          -
          <lpage>261</lpage>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          <string-name>
            <surname>Ericsson</surname>
          </string-name>
          , “Technology for Good,” Available online: https://www.ericsson.com/assets/local/aboutericsson/sustainability-and-corporateresponsibility/documents/2015-corporateresponsibility-and
          <string-name>
            <surname>-</surname>
          </string-name>
          sustainability
          <source>-report.pdf Accessed on 30-11-2017</source>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          <string-name>
            <given-names>C.</given-names>
            <surname>Calero</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Piattini</surname>
          </string-name>
          , “
          <article-title>Introduction to Green in software engineering,” Green Softw</article-title>
          . Eng., pp.
          <fpage>1</fpage>
          -
          <lpage>327</lpage>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          <string-name>
            <given-names>N.</given-names>
            <surname>Condori-Fernandez</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G.</given-names>
            <surname>Procaccianti</surname>
          </string-name>
          , and
          <string-name>
            <given-names>N.</given-names>
            <surname>Ali</surname>
          </string-name>
          , “
          <article-title>Metrics for green and sustainable software</article-title>
          :
          <source>MeGSuS</source>
          <year>2014</year>
          ,” in Proceedings - 2014
          <source>Joint Conference of the International Workshop on Software Measurement, IWSM 2014 and the International Conference on Software Process and Product Measurement</source>
          ,
          <year>Mensura 2014</year>
          ,
          <year>2014</year>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          and
          <string-name>
            <given-names>A.</given-names>
            <surname>Fleischmann</surname>
          </string-name>
          , “Teach Sustainability in Software Engineering?,
          <source>” in 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&amp;T)</source>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          <string-name>
            <given-names>S.</given-names>
            <surname>Naumann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Dick</surname>
          </string-name>
          , E. Kern, and T. Johann, “
          <article-title>The GREENSOFT Model: A reference model for green and sustainable software and its engineering,”</article-title>
          <string-name>
            <given-names>Sustain. Comput. Informatics</given-names>
            <surname>Syst</surname>
          </string-name>
          ., vol.
          <volume>1</volume>
          , no.
          <issue>4</issue>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          <string-name>
            <surname>Richardson</surname>
          </string-name>
          , “
          <article-title>Developing a sustainability nonfunctional requirements framework</article-title>
          ,
          <source>” Proc. 3rd Int.</source>
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          <string-name>
            <given-names>Work. Green</given-names>
            <surname>Sustain. Softw</surname>
          </string-name>
          . -
          <source>GREENS</source>
          <year>2014</year>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          <string-name>
            <surname>C. C. Venters</surname>
          </string-name>
          et al., “
          <article-title>The Blind Men and the Elephant Towards an Empirical Evaluation Framework for Software Sustainability</article-title>
          ,” vol.
          <volume>2</volume>
          , no.
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          1, pp.
          <fpage>1</fpage>
          -
          <lpage>6</lpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          <string-name>
            <surname>Tomlinson</surname>
          </string-name>
          , “
          <article-title>Safety, security, now sustainability: The nonfunctional requirement for the 21st century,” IEEE Softw.</article-title>
          , vol.
          <volume>31</volume>
          , no.
          <issue>3</issue>
          , pp.
          <fpage>40</fpage>
          -
          <lpage>47</lpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          , “
          <source>Supporting Sustainability [23] [24] [25] Aspects in Software Engineering,” 3rd Int. Conf.</source>
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          <string-name>
            <surname>Comput. Sustain.</surname>
          </string-name>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>4</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          <string-name>
            <given-names>S.</given-names>
            <surname>Oyedeji</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Seffah</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          , “
          <article-title>A catalogue supporting software sustainability design,” Sustainability</article-title>
          , vol.
          <volume>10</volume>
          , no.
          <issue>7</issue>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>30</lpage>
          ,
          <year>2018</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          , “
          <article-title>Infusing green: Requirements engineering for green in and through software systems</article-title>
          ,
          <source>” 3rd Intl. Work. Requir. Eng. Sustain. Syst.</source>
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          <year>2014</year>
          , vol.
          <volume>1216</volume>
          , no.
          <issue>1</issue>
          , pp.
          <fpage>44</fpage>
          -
          <lpage>53</lpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          <string-name>
            <given-names>K.</given-names>
            <surname>Roher</surname>
          </string-name>
          and
          <string-name>
            <given-names>D.</given-names>
            <surname>Richardson</surname>
          </string-name>
          , “Sustainability requirement patterns,
          <source>” 2013 3rd Int. Work. Requir.</source>
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          <string-name>
            <surname>Patterns</surname>
          </string-name>
          ,
          <article-title>RePa 2013 - Proc</article-title>
          ., pp.
          <fpage>8</fpage>
          -
          <lpage>11</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          John Wiley &amp; Sons, Ltd,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          <string-name>
            <given-names>T.</given-names>
            <surname>Juha</surname>
          </string-name>
          , “Good, bad, and
          <article-title>beautiful software</article-title>
          .
          <source>In search of green software quality factors,” CEPIS Upgrad. XII 422-27</source>
          , no.
          <source>July</source>
          ,
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          <string-name>
            <given-names>L.</given-names>
            <surname>Erdmann Hilty</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            ,
            <surname>Goodman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Arnfalk</surname>
          </string-name>
          ,
          <string-name>
            <surname>P.</surname>
          </string-name>
          and
          <string-name>
            <given-names>L. Erdmann</given-names>
            <surname>Hilty</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            ,
            <surname>Goodman</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Arnfalk</surname>
          </string-name>
          ,
          <string-name>
            <surname>P.</surname>
          </string-name>
          , “The Future Impact of ICTs on Environmental Sustainability,” IPTS Publ., no.
          <source>August</source>
          , p.
          <fpage>68</fpage>
          ,
          <year>2004</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          <string-name>
            <given-names>M. N.</given-names>
            <surname>Malik</surname>
          </string-name>
          and
          <string-name>
            <given-names>H. H.</given-names>
            <surname>Khan</surname>
          </string-name>
          , “
          <article-title>Investigating Software Standards: A Lens of Sustainability for Software Crowdsourcing</article-title>
          ,” IEEE Access, vol.
          <volume>6</volume>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          <string-name>
            <given-names>Commun. Technol. Sustain. ETH</given-names>
            <surname>Zurich</surname>
          </string-name>
          ,
          <year>Febr</year>
          .
          <fpage>14</fpage>
          -
          <lpage>16</lpage>
          ,
          <year>2013</year>
          , no.
          <source>January</source>
          , pp.
          <fpage>175</fpage>
          -
          <lpage>182</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          <string-name>
            <given-names>K.</given-names>
            <surname>Erdélyi</surname>
          </string-name>
          , “
          <article-title>Special factors of development of green software supporting eco sustainability</article-title>
          ,
          <source>” SISY 2013 - IEEE 11th Int. Symp. Intell. Syst. Informatics, Proc.</source>
          , pp.
          <fpage>337</fpage>
          -
          <lpage>340</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          <string-name>
            <given-names>M.</given-names>
            <surname>Dick</surname>
          </string-name>
          and
          <string-name>
            <given-names>S.</given-names>
            <surname>Naumann</surname>
          </string-name>
          , “
          <article-title>Enhancing software engineering processes towards sustainable software product design</article-title>
          ,
          <source>” 24th Int. Conf. Informatics Environ. Prot. (EnviroInfo</source>
          <year>2010</year>
          ), vol.
          <year>2010</year>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          <string-name>
            <given-names>M.</given-names>
            <surname>Colmant</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.</given-names>
            <surname>Rouvoy</surname>
          </string-name>
          , and L. Seinturier, “
          <article-title>Improving the energy efficiency of software systems for multi-core architectures</article-title>
          ,
          <source>” Proc. 11th Middlew</source>
          . Dr. Symp.
          <article-title>MDS 2014 - co-located with ACM/IFIP/USENIX 15th Int</article-title>
          . Middlew. Conf., pp.
          <fpage>2</fpage>
          -
          <lpage>5</lpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          <string-name>
            <surname>i</surname>
          </string-name>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          et al.,
          <source>“ICT4S</source>
          <year>2029</year>
          ௗ
          <article-title>: What will be the Systems Supporting Sustainability in 15 Yearsௗ</article-title>
          ?,”
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          <string-name>
            <given-names>S.</given-names>
            <surname>Jansen</surname>
          </string-name>
          and
          <string-name>
            <given-names>M.</given-names>
            <surname>Cusumano</surname>
          </string-name>
          , “
          <article-title>Defining software ecosystems: a survey of software platforms</article-title>
          and
          <source>[26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38]</source>
          [39]
          <article-title>business network governanceௗ : Software Ecosystems Analyzing and Managing Business Networks in the Software Industry</article-title>
          ,” Softw. Ecosyst. Anal. Manag.
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          <string-name>
            <given-names>Bus. Networks</given-names>
            <surname>Softw</surname>
          </string-name>
          . Ind., pp.
          <fpage>13</fpage>
          -
          <lpage>28</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          <string-name>
            <surname>Awodele</surname>
          </string-name>
          , “Software Ecosystem: Features, Benefits and Challenges,”
          <string-name>
            <surname>Int</surname>
          </string-name>
          .
          <source>J. Adv. Comput. Sci. Appl</source>
          ., vol.
          <volume>4</volume>
          , no.
          <issue>8</issue>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>6</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          and
          <string-name>
            <given-names>H.</given-names>
            <surname>Femmer</surname>
          </string-name>
          , “
          <article-title>A generic model for sustainability with process- and product-specific instances</article-title>
          ,
          <source>” GIBSE 2013 - Proc</source>
          .
          <year>2013</year>
          Work. Green Softw. Eng.
          <article-title>Green by Softw</article-title>
          . Eng., no.
          <source>June</source>
          <year>2015</year>
          , pp.
          <fpage>3</fpage>
          -
          <lpage>7</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          <string-name>
            <given-names>E.</given-names>
            <surname>Britannica</surname>
          </string-name>
          , “
          <article-title>Measurement instruments and systems</article-title>
          .
          <source>Accessed on 7-12-2017</source>
          from: https://www.britannica.com/print/article/371701,” pp.
          <fpage>1</fpage>
          -
          <lpage>2</lpage>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          <string-name>
            <given-names>R.</given-names>
            <surname>Seacord</surname>
          </string-name>
          et al.,
          <source>“Measuring Software Sustainability,” J. Chem. Inf. Model.</source>
          , vol.
          <volume>53</volume>
          , no.
          <issue>9</issue>
          , pp.
          <fpage>1689</fpage>
          -
          <lpage>1699</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          <string-name>
            <given-names>F.</given-names>
            <surname>Albertao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Xiao</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Tian</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Y.</given-names>
            <surname>Lu</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K. Q.</given-names>
            <surname>Zhang</surname>
          </string-name>
          , and C. Liu, “
          <article-title>Measuring the Sustainability Performance of Software Projects,” 2010 IEEE 7th Int</article-title>
          . Conf. E-bus. Eng., pp.
          <fpage>369</fpage>
          -
          <lpage>373</lpage>
          ,
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          <string-name>
            <given-names>G.</given-names>
            <surname>Lami</surname>
          </string-name>
          and
          <string-name>
            <given-names>L.</given-names>
            <surname>Buglione</surname>
          </string-name>
          , “
          <article-title>Measuring software sustainability from a process-centric perspective</article-title>
          ,
          <source>” Proc. 2012 Jt. Conf. 22nd Int. Work. Softw. Meas.</source>
        </mixed-citation>
      </ref>
      <ref id="ref38">
        <mixed-citation>
          <source>2012 7th Int. Conf. Softw. Process Prod. Meas.</source>
        </mixed-citation>
      </ref>
      <ref id="ref39">
        <mixed-citation>
          <string-name>
            <surname>IWSM-MENSURA</surname>
          </string-name>
          <year>2012</year>
          , pp.
          <fpage>53</fpage>
          -
          <lpage>59</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref40">
        <mixed-citation>
          <string-name>
            <surname>M. R. Idio</surname>
          </string-name>
          , “Measuring Sustainability Impact of Software,” vol.
          <volume>16</volume>
          , no.
          <issue>1</issue>
          , pp.
          <fpage>5</fpage>
          -
          <lpage>7</lpage>
          ,
          <year>2014</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref41">
        <mixed-citation>
          <article-title>({GREENS)}, 2013 2nd Int</article-title>
          . Work., pp.
          <fpage>46</fpage>
          -
          <lpage>53</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref42">
        <mixed-citation>
          <string-name>
            <given-names>S.</given-names>
            <surname>Naumann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Dick</surname>
          </string-name>
          , E. Kern, and T. Johann, “
          <article-title>The GREENSOFT Model: A reference model for green and sustainable software and its engineering,”</article-title>
          <string-name>
            <given-names>Sustain. Comput. Informatics</given-names>
            <surname>Syst</surname>
          </string-name>
          ., vol.
          <volume>1</volume>
          , no.
          <issue>4</issue>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref43">
        <mixed-citation>
          <string-name>
            <given-names>S.</given-names>
            <surname>Oyedeji</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Seffah</surname>
          </string-name>
          , and
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          , “Sustainability Quantification in Requirements Informing Design,
          <source>” 6th Int. Work. Requir. Eng.</source>
        </mixed-citation>
      </ref>
      <ref id="ref44">
        <mixed-citation>
          <string-name>
            <surname>Sustain. Syst.</surname>
          </string-name>
          ,
          <source>vol. i</source>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref45">
        <mixed-citation>
          <string-name>
            <surname>K.-L. Kramer</surname>
          </string-name>
          ,
          <source>User Experience in the Age of Sustainability</source>
          .
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref46">
        <mixed-citation>
          <string-name>
            <surname>Seinturier</surname>
          </string-name>
          , “
          <article-title>Runtime monitoring of software energy hotspots</article-title>
          ,” in
          <source>Automated Software Engineering (ASE)</source>
          ,
          <source>2012 Proceedings of the 27th IEEE/ACM International Conference on, 2012</source>
          , pp.
          <fpage>160</fpage>
          -
          <lpage>169</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref47">
        <mixed-citation>
          <source>April</source>
          <year>2015</year>
          , pp.
          <fpage>51</fpage>
          -
          <lpage>54</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref48">
        <mixed-citation>
          <string-name>
            <surname>M. A. Ferreira</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          <string-name>
            <surname>Hoekstra</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Merkus</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          <string-name>
            <surname>Visser</surname>
            , and
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Visser</surname>
          </string-name>
          , “
          <article-title>Seflab: A lab for measuring software energy footprints</article-title>
          ,
          <source>” in 2013 2nd International Workshop on Green and Sustainable Software, GREENS 2013 - Proceedings</source>
          ,
          <year>2013</year>
          , pp.
          <fpage>30</fpage>
          -
          <lpage>37</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref49">
        <mixed-citation>
          <string-name>
            <given-names>V.</given-names>
            <surname>Cordero</surname>
          </string-name>
          ,
          <string-name>
            <given-names>I. G.</given-names>
            <surname>R. De Guzmán</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Piattini</surname>
          </string-name>
          , “
          <article-title>A first approach on legacy system energy consumption measurement</article-title>
          ,”
          <source>in Proceedings - 2015 IEEE 10th International Conference on Global Software Engineering Workshops, ICGSEW</source>
          <year>2015</year>
          ,
          <year>2015</year>
          , pp.
          <fpage>35</fpage>
          -
          <lpage>43</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref50">
        <mixed-citation>
          <string-name>
            <given-names>S.</given-names>
            <surname>Cagri</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Furkan</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>James</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Fouad</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.</given-names>
            <surname>Lori</surname>
          </string-name>
          , and W. Kristina, “Towards Power Reduction Through Improved Software Design,” pp.
          <fpage>1</fpage>
          -
          <lpage>8</lpage>
          ,
          <year>2007</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref51">
        <mixed-citation>
          <string-name>
            <given-names>P.</given-names>
            <surname>Bozzelli</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Gu</surname>
          </string-name>
          , and
          <string-name>
            <given-names>P.</given-names>
            <surname>Lago</surname>
          </string-name>
          , “
          <article-title>A systematic literature review on green software metrics</article-title>
          ,
          <source>” Sis.Uta.Fi</source>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref52">
        <mixed-citation>
          <string-name>
            <surname>Johann</surname>
          </string-name>
          , “Green Software and
          <string-name>
            <surname>Green Software</surname>
          </string-name>
          Engineering - Definitions , Measurements , and Quality Aspects,” pp.
          <fpage>87</fpage>
          -
          <lpage>94</lpage>
          ,
          <year>2013</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref53">
        <mixed-citation>
          <string-name>
            <given-names>T.</given-names>
            <surname>Debbarma</surname>
          </string-name>
          and
          <string-name>
            <given-names>K.</given-names>
            <surname>Chandrasekaran</surname>
          </string-name>
          , “
          <article-title>Green measurement metrics towards a sustainable software: A systematic literature review</article-title>
          ,
          <source>” 2016 Int.</source>
        </mixed-citation>
      </ref>
      <ref id="ref54">
        <mixed-citation>
          <string-name>
            <given-names>Conf. Recent</given-names>
            <surname>Adv</surname>
          </string-name>
          .
          <source>Innov. Eng. ICRAIE</source>
          <year>2016</year>
          ,
          <year>2017</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref55">
        <mixed-citation>
          155, no.
          <source>June</source>
          .
          <year>2011</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref56">
        <mixed-citation>
          <string-name>
            <given-names>T.</given-names>
            <surname>Johann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Dick</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Naumann</surname>
          </string-name>
          , and E. Kern, “
          <article-title>How to measure energy-efficiency of software: Metrics and measurement results,” 2012 1st Int</article-title>
          .
        </mixed-citation>
      </ref>
      <ref id="ref57">
        <mixed-citation>
          <string-name>
            <given-names>Work. Green</given-names>
            <surname>Sustain</surname>
          </string-name>
          .
          <source>Software, GREENS 2012 - Proc.</source>
          , pp.
          <fpage>51</fpage>
          -
          <lpage>54</lpage>
          ,
          <year>2012</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref58">
        <mixed-citation>
          <string-name>
            <given-names>G.</given-names>
            <surname>Oleksandr</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Vyacheslav</surname>
          </string-name>
          , and
          <string-name>
            <given-names>F.</given-names>
            <surname>Mario</surname>
          </string-name>
          , “
          <source>Software Quality Standards and Models Evolution: Greenness</source>
          and
          <string-name>
            <given-names>Reliability</given-names>
            <surname>Issues</surname>
          </string-name>
          ,” vol.
          <volume>469</volume>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref59">
        <mixed-citation>
          <string-name>
            <surname>Lepage</surname>
            ,
            <given-names>and J.</given-names>
          </string-name>
          <string-name>
            <surname>Georges</surname>
          </string-name>
          , “Measurements and Sustainability,”
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref60">
        <mixed-citation>
          <string-name>
            <given-names>S. A. .</given-names>
            <surname>Koçak</surname>
          </string-name>
          ,
          <string-name>
            <given-names>G. I. .</given-names>
            <surname>Alptekin</surname>
          </string-name>
          ,
          <article-title>and</article-title>
          <string-name>
            <surname>A. B. . Bener,</surname>
          </string-name>
          “
          <article-title>Evaluation of software product quality attributes and environmental attributes using ANP decision framework,” CEUR Workshop Proc</article-title>
          ., vol.
          <volume>1216</volume>
          , pp.
        </mixed-citation>
      </ref>
      <ref id="ref61">
        <mixed-citation>
          <string-name>
            <given-names>S.</given-names>
            <surname>Abdulsalam</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Z.</given-names>
            <surname>Zong</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Q.</given-names>
            <surname>Gu</surname>
          </string-name>
          , and
          <string-name>
            <given-names>M.</given-names>
            <surname>Qiu</surname>
          </string-name>
          , “
          <article-title>Using the Greenup, Powerup, and Speedup metrics to evaluate software energy efficiency</article-title>
          ,
          <source>” 2015 6th Int.</source>
        </mixed-citation>
      </ref>
      <ref id="ref62">
        <mixed-citation>
          <string-name>
            <given-names>Green</given-names>
            <surname>Sustain</surname>
          </string-name>
          .
          <source>Comput. Conf.</source>
          ,
          <year>2016</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref63">
        <mixed-citation>
          <string-name>
            <given-names>A.</given-names>
            <surname>Michel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Attali</surname>
          </string-name>
          , and E. Bush, “
          <article-title>Energy efficiency of White Goods in Europeௗ : monitoring the market with sales data,” no</article-title>
          . June, pp.
          <fpage>1</fpage>
          -
          <lpage>53</lpage>
          ,
          <year>2015</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>