<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Social sustainability by design: Experiences with the Sustainable Awareness Framework ⋆</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>John Krogstie</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Dimitra Chasanidou</string-name>
          <email>Dimitra.Chasanidou@ntnu.no</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Birgit Krogstie</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Norwegian University of Science and Technology, NTNU</institution>
        </aff>
      </contrib-group>
      <abstract>
        <p>Several techniques have been developed over the years for capturing sustainability requirements including capturing social and personal sustainability issues e.g. related to discuss ethics and values in connection to software solutions. One of these techniques is the Sustainability Awareness Framework (SusAF) which covers both environmental, social, individual, technical, and economic sustainability. We have used this technique in both industrial and academic settings, and in this paper, we will share some of the experiences with the technique and point to improvements. Better tool and method support is perceived useful to be able to have more detailed analysis of different aspects of sustainability and follow this up in software development over a longer period of time.</p>
      </abstract>
      <kwd-group>
        <kwd>Sustainability requirements</kwd>
        <kwd>Sustainability awareness</kwd>
        <kwd>Method evaluation1</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Information and Communication Technology (ICT) plays an important role in assuring both
social, environmental and economic sustainability. The need for the ICT field to address
sustainability has been acknowledged for a long time in areas such as information systems [
        <xref ref-type="bibr" rid="ref17">17</xref>
        ],
Human-Computer Interaction (HCI), and software engineering [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. The impact of ICT can be
seen as both direct and indirect effects of the software and hardware [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ]. Direct effects such as
energy consumption are what Hilty et al. [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ] denote first-order effects. Second-order effects
include the consequences of processes being changed by the application of ICT. Third-order
effects are seen as long- and medium-term change in behavior, such as change in consumption
patterns, and change in economic structures. The use of new technologies such as AI and IoT
can be important for creating positive secondary effects to address sustainability challenges
given we are able to use new technologies sufficiently rapid for potentially achieving positive
effects [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ], but can have even larger negative first-order, second-order and third-order effects
if used wrongly. The information system and software development processes are the most
important enablers for a future where ethical and trustworthy IT solutions impact the quality
of people's lives. Today, the processes for designing and evaluating software are primarily based
on direct functionality, cost and monetary value for industry, without sufficient focus on the
wider societal and environmental impact. A shift towards a focus on sustainable development
constitutes a major change in perspective. In software engineering and other sectors, there has
been some focus on the environmental dimension of sustainability, evidenced by [
        <xref ref-type="bibr" rid="ref2 ref7">2,7</xref>
        ].
However, there is a growing need to investigate the social dimension, combined with the
individual and human dimensions, to achieve sustainability in software engineering [
        <xref ref-type="bibr" rid="ref11 ref4 ref6">4,6,11</xref>
        ].
      </p>
      <p>
        As seen in [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ], there are several approaches to sustainability by design, looking at
sustainability concerns already in the requirements phase. One of these, known as Sustainable
Awareness Framework (SusAF) has been developed by Becker et al [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] and is structuring the
effects of software systems into five dimensions. Three of them have been used in several
sustainability models since the eighties: The economic (monetary), the environmental, and the
social dimensions. To this, Becker and colleagues add the individual and the technical
dimension. In a review study of techniques to take sustainability into account in requirements
engineering, SusAF is the most discussed approach, but these are publications from the team
who developed the framework, beyond what is described in this literature, the application of
the framework is limited [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ].
      </p>
      <p>In section 2 we will present SusAF in detail. In section 3, we elaborate on the method
including the usage of the method in different settings. In section 4 we discuss experiences from
the use and point to both positive aspects and areas with room for improvement. Section 5
concludes the paper, pointing to further work.</p>
    </sec>
    <sec id="sec-2">
      <title>2. Background on SusAF</title>
      <p>The five dimensions of sustainability used in SusAF are often depicted as a pentagon as seen
in Figure 1 below. For each of the five dimensions, the model distinguishes between
immediate, enabling and structural effects, corresponding to the first-, second- and
thirdorder effects described briefly above.</p>
      <p>
        This pentagon model has been used in the development of a model for sustainability
evaluation of ICT projects [
        <xref ref-type="bibr" rid="ref13 ref4">4,13</xref>
        ] where it is denoted as ‘sustainability analysis diagram’
(SusAD). Specifically, SusAF is composed of the SusAD, the chart for visualizing potential
sustainability effects of a software system, and five question sets for guiding exploration of
the five sustainability dimensions [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ]. Other frameworks combine the personal and social
levels into one, having four dimensions [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ].
      </p>
      <p>The model discussing this from a software engineering point of view.</p>
      <p>
        The five dimensions can be described in more detail in the following way [
        <xref ref-type="bibr" rid="ref13 ref4">4,13</xref>
        ]:
•
•
•
•
•
      </p>
      <p>The environmental dimension covers the use and stewardship of natural resources.
It includes material and resources, waste and pollution, biodiversity, energy and
logistics.</p>
      <p>The technical dimension covers the ability to maintain and evolve artificial systems
(such as software) over time, including aspects of maintainability, usability,
adaptability, security, and scalability.</p>
      <p>The individual dimension covers health, lifelong learning, privacy, safety, and
agency.</p>
      <p>The social dimension covers relationships between individuals and groups. This
includes sense of community, trust, inclusiveness and diversity, equity,
participation, and communication.</p>
      <p>The economic dimension covers financial value, customer relationship
management, supply chain, governance and innovation.</p>
      <p>Aspects within these dimensions are often interlinked, so that an effect in one area can
have a positive or negative effect on another effect in the same or different dimension, on
another level.</p>
      <p>A standard technique has been developed for the usage of this framework to gather
requirements for doing analysis of a setting in a facilitated manner. A workbook describes
this process accommodated to different timings typically between 2 and 4 hours according
to the process depicted in Figure 2.</p>
      <p>An important part of the framework in addition to the pentagon diagram and the process,
is a set of driving questions within each subarea of the dimensions as described above
(privacy, safety, energy etc.).</p>
    </sec>
    <sec id="sec-3">
      <title>3. Selected experiences</title>
      <p>We have used the SusAF framework in many settings, both within academia in connection with
courses, in research and in industrial projects, and in workshops together with practitioners.
The technique is meant to be generally useful, thus we find it interesting to bring together
experiences from quite different usage settings.</p>
      <p>The technique has typically been used in facilitated workshops, in some cases with a
followup. The experience reported from here are from the following cases:
•
•
•
•</p>
      <sec id="sec-3-1">
        <title>Redesign of a mobile augmented reality system (MAR) [5]. A 4-hour workshop at a practitioner conference on ICT in public sector. Used as part of an HCI- course on university-level. Used as part of a service design process used in a course on digital innovation.</title>
        <sec id="sec-3-1-1">
          <title>3.1. Mobile Augmented Reality (MAR)-project</title>
          <p>Description of setting: The aim of the project was to redesign a MAR application with a focus
on sustainability design. The motivation for re-designing was the poor technical sustainability
in third-party service and therefore, sustainability design was chosen as an integrated approach
for redesign. A workshop was arranged. The objective of the workshop was to discuss the app’s
sustainability issues and envision the development of the app. Five participants who were
involved in the previous version of the app joined the workshop. Participants were mostly
developers, and one participant had a design background.</p>
          <p>Detailed description of how the technique was used: Before the workshop, previous work
in the field of MAR travel guide apps was searched to create a potential list of sustainability
issues and topics that could be relevant discussion topics during the workshop. Then, a
workshop was organized to envision the sustainability of the system within a decade and to
capture chains of effects. During the workshop, two researchers together with participants,
worked on a mapping exercise based on SusAF. For that purpose, information material about
the technique was sent to participants beforehand. The workshop started with a brief
introduction to the project and the identified sustainability issues, followed by an introduction
to SusAF. Roles were assigned, where the participants were responsible to discuss one
dimension each of SusAF but could contribute to all. Then, participants were given three tasks:
a mapping exercise, levels of effects and chain of effects. In all tasks, participants worked first
individually and then in the group. In addition, driving questions were used to spark
discussions, or redirect when it was needed. The workshop lasted approximately 2 hours and,
at the end, facilitators concluded by summarizing findings and discussing experiences. Finally,
sustainability issues with immediate effects were selected for further development into design
suggestions for an MAR app.</p>
          <p>Description of learning related to the use of technique: Overall, participants reported
positive experience with SusAF. The stepwise process of the tool gave participants a systematic
and integrated approach to work with sustainability design, with guidance through specific
tasks to follow and a clear sequence of steps, ensuring that all relevant aspects and dimensions
of sustainability are addressed. Especially for the extraction of design suggestions, it allowed
formulation of requirements that were easily translated into design suggestions. Regarding the
teamwork, SusAF allowed multidisciplinary view on sustainability and role-playing activity for
participants.</p>
          <p>There were also some challenges associated with the use of SusAF. First, there is a lack of
collective understanding of what sustainability means for software development and how to
identify sustainability issues. Identifying such issues and mapping them to dimensions is not an
easy and straightforward task. It requires preparation, research and benchmarking existing
MAR apps or similar technologies to identify common sustainability challenges and best
practices. Furthermore, applying SusAF demands good understanding of the framework and its
application, which may require specialized knowledge or training. Finally, translating
sustainability issues into design suggestions is an iterative and ongoing process, where a deep
understanding of sustainability issues, user needs, and the application context are needed.</p>
        </sec>
        <sec id="sec-3-1-2">
          <title>3.2. Workshop at practitioner conference</title>
          <p>Description of setting: Practitioners in public sector and IT industry who did not know each
other in advance signed up to one of ten parallel workshops at a conference. They worked
physically with different colored post-its on a pentagon-poster, and with a flip-over to support
the prioritization of ideas. The participants worked on a partly artificial case (i.e., the case was
real but was not from the participants themselves).</p>
          <p>Detailed description of how the technique was used: The workshop lasted four hours. An
one hour introduction to sustainability issues and SusAF was followed by a three-hour
workshop, including a 30-minute break and a 30 minute wrap-up. The workshop followed the
prescribed process: The case was introduced. Each participant devised 2-3 impacts per
dimension individually, with different colors for positive and negative effects. Then all points
were discussed and prioritized by the whole group. Points that were regarded as high priority
were put on the A1 SusAF board as illustrated in Figure 3. T hen the different effects were
connected wherever relations were identified. Two facilitators helped.</p>
          <p>Description of learning related to the use of technique: Not knowing each other at the
outset, it took some time for the participants to become comfortable. In such setting it is often
useful to work on a situation not being from one of the organizations. It was an advantage to
do the prioritization i.e. not put everything on the pentagon from the start. Still the middle of
the chart became cluttered, and it was hard to model the relationships on the board. That the
pentagon looks like a spider diagram also made some participants a bit confused in the
beginning. Participants gave positive feedback on the usefulness of the technique but did not
feel comfortable about running such workshops themselves.</p>
        </sec>
        <sec id="sec-3-1-3">
          <title>3.3. Workshop with students as part of a course</title>
          <p>Description of setting: The setting was a mandatory 7.5 ECTS course on CSCW and user
centered design for 1st year students in a business-oriented IT bachelor program. The course
included a set of assignments together comprising a small design project where groups of 4-5
students worked on a solution to be implemented in Sharepoint.</p>
          <p>The overall objective from an educational perspective was to enrich the course with learning
activities that would contribute to achievement of the existing learning outcomes. Including a
SusAF analysis in the projects was to help the students become more aware of sustainability
effects of their proposed solutions and acquire skills in identifying sustainability effects in other
settings.</p>
          <p>Detailed description of how the technique was used: At the time of using SusAF, the
students had started work on personas and scenarios for their design projects. A half-day
facilitated SusAF workshop was provided within normal lecture time in the course. Attendance
was strongly encouraged, results from the workshop being part of an assignment to be handed
in, and close to everyone (ca. 40 students) in class attended. In the workshop, an introduction
to the framework was given by the instructor, including examples referring to their case. Next,
the students worked in teams to develop SusAD diagrams for their own solutions. Key
adaptations of the process include:
•
•
•</p>
        </sec>
      </sec>
      <sec id="sec-3-2">
        <title>Students were working in a co-located manner in one large room. For the diagram, A1 cardboard templates, post-it notes and pens were used. The five sets of driving questions were distributed to all groups to help them identify sustainability effects.</title>
        <p>•
•
•</p>
        <p>Instructions were given that the groups might choose two or three main types of
functionality/features of their design and make one diagram for each.</p>
        <p>Presentation of the results in were made in clusters of groups (3 groups presenting
to each other).</p>
        <p>Evaluation of the use of SusAF was covered indirectly as workshop results were part
of an assignment and as students reflected on the course activities as part of the
summative evaluation (pass/fail) in the course.</p>
        <p>Description of learning related to the use of technique: The task and technique were easy
to understand for the students. Only limited facilitation of the groupwork was needed. Having
a similar (although not identical) case among the participating groups made it possible for the
instructor to create examples that all participants could easily relate to, making the introduction
to the technique efficient and effective. The sustainability analysis was considered meaningful
considering the overall learning outcomes and activities in the course. The intervention
demonstrated that it is possible to enrich a curriculum with insights on sustainability by use of
SusAF without necessarily making resource-consuming changes to a course and without
creating the feeling of sustainability focus being enforced top-down and/or in a superficial way.
The students reported getting new insights/ideas from other groups presenting their diagrams.
Focusing on distinct aspects/functionalities/features in separate diagrams helped avoid
cluttering of the diagrams (but at the known expense of not being able to make connections
between sustainability effects across the diagrams).</p>
        <sec id="sec-3-2-1">
          <title>3.4. Used in service design in a multidisciplinary course on digital innovation.</title>
          <p>
            Description of setting: SusAF was used in connection to a service design process following a
double diamond approach [
            <xref ref-type="bibr" rid="ref10">10</xref>
            ]. It was used two times, first on the problem statement, and then
on the final prototype solution to see if it had to be updated when the group knew more about
the technical solution. The process was part of a cross-disciplinary course for bachelor students
where together they were meant to provide input to a digital innovation using XR-technology.
Detailed description of how the technique was used: The workshop was held in 2 hours
(which is less than normal). The week before, sustainability and selected techniques such as
SusAF had been presented. After a short introduction to the framework, the students worked
in their established cross disciplinary groups for two hours. The physical setting was using
different colored Post-its on an A1 poster showing the pentagon. The students worked with one
idea at the time, and each student focused on one dimension. Invariably and archetypically, in
each group it was the IT-student who was put to work on the technical dimension. Based on
the driving question each participant came up with issues and presented them to the others
(one at a time). After a lunch break, the students discussed how issues were related, and
identified issues to have in mind in the further work. The groups then presented their findings
to the other groups.
          </p>
          <p>The second part was to take up the first analysis and see to what extent it was changed based
on new knowledge achieved, while developing and evaluating a prototype, and report changes
in the sustainability analysis in the final project report 2 months after.</p>
          <p>Description of learning related to the use of technique: The students were quickly able to
start using the techniques on their initial ideas. They had discussed their ideas up front and
were at the time quite familiar with each other. Having a lecture on the technique before using
it was also useful. We witnessed at least one group learning during the project as demonstrated
by the final prototype including aspects not in the original idea. These were social and
individual aspects. A tool for documenting the results in a more persistent way would have
been useful, according to the participants. In the case, the results were documented by taking
pictures of the board.</p>
        </sec>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>4. Learnings and suggestion for improvements</title>
      <p>In general, those who work with SusAF for the first time find it possible to use the technique
after a relatively short introduction on sustainability and the framework. Many are
positively surprised, especially to find that it not only about environmental sustainability,
but also covers the social and individual aspects. It is important, though, that the first usage
is facilitated. Below is a summary of possible improvements.</p>
      <p>
        The workshop-version of SusAF available on the web [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ] provides one workshop
setting/timing. In practice (and due to different constraints) we find it useful to use different
variants of this, based on e.g., how well people know each other before the workshop,
importance of results, and what is the main objectives (develop software requirements,
learning about sustainability…).
      </p>
      <p>
        Participants tend to be satisfied with the outcome of a SusAF workshop, having
collaboratively identified relevant sustainability effects and seen connections between
them. However, to get the most out of the analysis in the longer term, we see a need for
guidelines for the long-term usage of the results. There is existing research on e.g., how to
bring SusAF into agile development, but this must be further experimented with [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
Another approach is to set aside time after the workshop to further analyze the situation,
and then return to another session of updating the model, as we did in the service design
case and in the MAR-case and using the result from the last workshop for developing
quantifiable requirements.
      </p>
      <p>As found in the MAR-project, it is often necessary with preparatory work before the
workshop, both for facilitators to understand the domain, and for the participants to quickly
start using the approach when at the workshop.</p>
      <p>
        The technique is primarily manual to be used in a physical workshop setting. This can
provide a good process to brainstorm and come up with initial ideas. When many ideas
come up it is however not easy to work with the diagram, and to bring forward the results
e.g., to design activities. Better tool support is needed, for the immediate analysis, for
bringing the result further, and for adding more detail, including assumptions for the issues
raised. This tool support should be linked to a proper modelling method. In the literature,
e.g. SAF [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ] provides a modeling method and tool to support it. A SusAF Miro-canvas exists,
but it is not very advanced considering the modeling one typically does in requirements
engineering and design. Another advantage of better tool support is that it will make it
easier to discuss the quality of the results produced, e.g. by adapting general frameworks
for evaluating the quality of models [
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
      <p>
        The pentagon-shape itself has some challenges. In our cases we have seen that it gave
some participants the erroneous impression that it is a spider-diagram. Another challenge
of the diagram layout is that most of the ideas are on the immediate level, which offers the
most limited space. This quickly leads to cluttering even when concerns are prioritized
before being put into the board. This problem might be addressed by simply changing the
proportions in the diagram, and with better tool support. A general concern regarding the
use of digital tools is that their use should not get in the way of the modeling process of the
humans [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ]. A key affordance of the pentagon diagram is that it facilitates low-threshold
collaboration between stakeholders who do not need to possess advanced knowledge of
digital tools. Appropriate modelling tools should maintain this advantage. There are also
some challenges with the proposed list of topics for each dimension. Some of them, such as
privacy often being a subset of security, go across several dimensions, which can be
confusing. On the other hand, since the topics are only proposed to drive the process and
help the participants not to forget areas, this is not so problematic. A bigger challenge is
what we regard as wrong placement of topics e.g. topics linked to dimensions where they
arguably do not belong. For instance, in our view usability is primarily a means to improve
the personal dimension, not the technical one. There are also missing areas, e.g., universal
design as a concern on the individual level. On this basis, we see that the topic lists could be
updated. It should be noted at this point that the authors of the SusAF framework do not
claim that their proposed topic list is exhaustive.
      </p>
      <p>
        It is challenging at times to decide the level of a sustainability effect, and more support
for the appropriate selection of level should be provided. In [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ] a fourth level is suggested,
which might make it easier to be clear on the level to use. At the same time a fourth level
introduces a risk of making the technique more complex.
      </p>
    </sec>
    <sec id="sec-5">
      <title>5. Conclusion and further work</title>
      <p>
        The SusAF framework has been used in several projects and project settings. Many pilots of the
use of the technique have been conducted by us and others. There is now a need for studies that
address the use of the technique in more detail and considers the process over longer time, e.g.,
how to support the development and evolution of (IT)-systems. In an educational context,
learning outcomes (e.g. related to discipline knowledge or sustainability knowledge) should be
evaluated after a certain time period. Also, one should look at support for doing more systematic
evaluation using e.g., MAM – Method Acceptance Model [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ]. We should also evaluate the
quality of the result. The resulting SusAD is a model that one can evaluate using e.g., SEQUAL
[
        <xref ref-type="bibr" rid="ref9">9</xref>
        ].
      </p>
      <p>This paper provides early results on the experiences of use of SusAF, collected in an informal
way. In later project and usage situation, we aim to follow the use more systematically, to have
even better grounding for our proposals to update the technique.</p>
    </sec>
    <sec id="sec-6">
      <title>Acknowledgements</title>
      <p>Thanks to all participants and organizers in the workshops reported upon in this paper.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>P.</given-names>
            <surname>Bambazek</surname>
          </string-name>
          , I. Groher,
          <string-name>
            <given-names>N.</given-names>
            <surname>Seyff</surname>
          </string-name>
          ,
          <article-title>Sustainability in Agile Software Development: A Survey Study among Practitioners</article-title>
          .
          <source>In 2022 International Conference on ICT for Sustainability (ICT4S)</source>
          (pp.
          <fpage>13</fpage>
          -
          <lpage>23</lpage>
          ). IEEE (
          <year>2022</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>P.</given-names>
            <surname>Bambazek</surname>
          </string-name>
          , I. Groher,
          <string-name>
            <given-names>N.</given-names>
            <surname>Seyff</surname>
          </string-name>
          ,
          <article-title>Requirements engineering for sustainable software systems: a systematic mapping study</article-title>
          ,
          <source>Requirements Engineering</source>
          , pp.
          <fpage>1</fpage>
          -
          <lpage>25</lpage>
          , (
          <year>2023</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>C.</given-names>
            <surname>Becker</surname>
          </string-name>
          , et. al.,
          <article-title>The Karlskrona manifesto for sustainability design</article-title>
          .
          <source>arXiv:1410</source>
          .
          <fpage>6968</fpage>
          .
          <string-name>
            <surname> </surname>
          </string-name>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>C.</given-names>
            <surname>Becker</surname>
          </string-name>
          et al.,
          <article-title>Requirements: The Key to Sustainability</article-title>
          . IEEE Softw., vol.
          <year>2016</year>
          , no. January/February, (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>D.</given-names>
            <surname>Chasanidou</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Krogstie</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            <surname>Boletsis</surname>
          </string-name>
          ,
          <article-title>Sustainability design in mobile augmented reality, in Norsk IKT-konferanse for forskning og utdanning</article-title>
          ,
          <source>no. 2</source>
          , (
          <year>2023</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <surname>C. M. de Souza</surname>
            ,
            <given-names>D. Soares</given-names>
          </string-name>
          <string-name>
            <surname>Cruzes</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          <string-name>
            <surname>Jaccheri</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          <string-name>
            <surname>Krogstie</surname>
          </string-name>
          ,
          <article-title>Social sustainability approaches for software development: A systematic literature review</article-title>
          ,
          <source>in International Conference on Product-Focused Software Process Improvement</source>
          , pp.
          <fpage>478</fpage>
          -
          <lpage>494</lpage>
          , Springer, (
          <year>2023</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>C.</given-names>
            <surname>Freitag</surname>
          </string-name>
          et al.
          <article-title>The real climate and transformative impact of ICT: A critique of estimates, trends, and regulations</article-title>
          .
          <source>Patterns</source>
          <volume>2</volume>
          ,
          <issue>9</issue>
          (
          <year>2021</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>L. M.</given-names>
            <surname>Hilty</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Aebischer</surname>
          </string-name>
          ,
          <article-title>ICT for Sustainability: An Emerging Research Field</article-title>
          . In: Hilty,
          <string-name>
            <given-names>L.M.</given-names>
            ,
            <surname>Aebischer</surname>
          </string-name>
          ,
          <string-name>
            <surname>B</surname>
          </string-name>
          . (eds.)
          <article-title>ICT Innovations for Sustainability</article-title>
          .
          <source>Advances in Intelligent Systems and Computing 310</source>
          , Springer International Publishing (
          <year>2015</year>
          ) 
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>J.</given-names>
            <surname>Krogstie</surname>
          </string-name>
          ,
          <source>Model based development and evolution of Information Systems. A quality Approach</source>
          . Springer (
          <year>2012</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>J.</given-names>
            <surname>Krogstie</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Sommerfeldt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. L.</given-names>
            <surname>Riise</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            <surname>Berge</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Fjeldvaer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Bjørnhaug</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L. S.</given-names>
            <surname>Flak</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            <surname>Håmo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A. H.</given-names>
            <surname>Vik</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M. Hole</given-names>
            <surname>Skogen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P. J. Virmalainen</given-names>
            <surname>Jøsendal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            <surname>Krogstie</surname>
          </string-name>
          ,
          <article-title>Development of a Toolbox on Sustainable ICT across Industry and Academia: The goforIT project</article-title>
          .
          <source>Proceedings Nokobit</source>
          , (
          <year>2023</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>P.</given-names>
            <surname>Lago</surname>
          </string-name>
          et. al.
          <article-title>Designing for Sustainability: Lessons Learned from Four Industrial Projects (</article-title>
          <year>2020</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <surname>D</surname>
            . L. Moody, G. Sindre,
            <given-names>T.</given-names>
          </string-name>
          <string-name>
            <surname>Brasethvik</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          <string-name>
            <surname>Sølvberg</surname>
          </string-name>
          ,
          <article-title>Evaluating the quality of process models: Empirical testing of a quality framework</article-title>
          .
          <source>Conceptual</source>
          Modeling-2002
          <string-name>
            <surname>:</surname>
          </string-name>
          (
          <year>2003</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>B.</given-names>
            <surname>Penzenstadler</surname>
          </string-name>
          et al. The SusA Workshop -
          <article-title>improving sustainability awareness to inform future business process and systems design https://zenodo</article-title>
          .org/records/3676514#.
          <source>YzFa9C8eOk7 last visited 18.2</source>
          .2024
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>A.</given-names>
            <surname>Rugeviciute</surname>
          </string-name>
          ,
          <string-name>
            <given-names>V.</given-names>
            <surname>Courboulay</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.M.</given-names>
            <surname>Hilty</surname>
          </string-name>
          ,
          <article-title>The research landscape of ICT for sustainability ICT4S (</article-title>
          <year>2023</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>J.</given-names>
            <surname>Stirna</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Persson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            <surname>Sandkuhl</surname>
          </string-name>
          , Participative Enterprise Modeling:
          <article-title>Experiences and Recommendations</article-title>
          . Proceeding CAiSE'
          <year>2007</year>
          (
          <year>2007</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <given-names>R.</given-names>
            <surname>Vinuesa</surname>
          </string-name>
          et. al.
          <source>The Role of Artificial Intelligence in Achieving the Sustainable Development  Goals. Nature Communications</source>
          <volume>11</volume>
          ,
          <fpage>233</fpage>
          . (
          <year>2020</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          [17]
          <string-name>
            <surname>J. vom Brocke</surname>
          </string-name>
          , R. T. Watson,
          <string-name>
            <given-names>C.</given-names>
            <surname>Dwyer</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Elliot</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.</given-names>
            <surname>Melville</surname>
          </string-name>
          ,
          <article-title>Green Information Systems: Directives for the IS Discipline</article-title>
          .
          <source>Communications of the Association for Information Systems</source>
          ,
          <volume>33</volume>
          , pp-pp.
          <fpage>https</fpage>
          ://doi.org/10.17705/1CAIS.03330 (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>