The Common Impact Data Standard (CIDS) is an ontology designed to represent a Social Purpose Organization's (SPO) impact model (i.e., definition) and the impact (i.e., effect) their implementation has on its stakeholders. It provides a common representation that allows each SPO to flexibly design an impact model that is most relevant to it, and report on its performance. CIDS spans the six dimensions of impact: What (Outcome, Impact), Who (Stakeholders), How (Program, Service, Activity), How Much (i.e., Indicator), Contribution (i.e., ImpactScale, ImpactDepth, ImpactDuration) and Risk (ImpactRisk). CIDS has been evaluated by members of the Common Approach project comprised of over 50 SPOs, grantmakers, etc., and has been implemented in several impact reporting commercial software tools.
allowing each SPO flexibility to design an impact model that is most relevant to it. The benefits of this flexible standard are: 1. Better impact: Each organization makes some difference, but their most impactful stories are when the data can be connected and aggregated. A common impact data standard allows networks to pool data, to see impact and use the data to improve impact. 2. Sophisticated analysis. CIDS makes it possible for researchers to integrate their data thereby enabling a plethora of analysis, e.g., longitudinal and transversal studies, using a variety of methods. This may lead to better understandings of needs, and a better understanding of what works. 3. More autonomy. Donors, investors, government agencies are increasingly aware that old impact reporting techniques have been a burden to grantees and investors. A common impact data model provides funders the standard formats that they need to understand portfolio-level impacts, while leaving SPOs the autonomy to measure impact in ways that best-fit the SPOs own data needs. 4. Less paperwork: A common impact data model allows impact data to be represented in ways that can accommodate the reporting needs of diverse funders. SPOs a common impact data model will need to do less custom reporting. 5. Greater visibility: Enable the tagging of an organization’s content on the internet making it easier for search engine users to find impact content on the web. 6. More versatility: A common data model makes it easier for organizations to connect their impact measurement with other measurement standards, such as the UN SDG Global Indicator Framework, IRIS+ and the International Aid Transparency Initiative (IATI) Standard.
This paper reports on the development, evaluation and deployment of the Common Impact Data Standard (CIDS). CIDS is an ontology defined using Description Logic and published in OWL4. We begin with a review of the Impact Management Project5 framework which provides a basis and use case for the development of CIDS. We then describe some of the ontology patterns contained in CIDS. We follow this with a description of the various evaluations performed, and its ongoing adoption in the social impact sector.
Impact measurement experts have developed numerous Impact Models to help social purpose organizations to articulate the change they seek to achieve and how that change is achieved. These Impact models include the Logic Model (PCI, 1979), Theory of Change (Weiss, 1997), Outcome Map (Earl et al., 2001), Outcome Chain (Harries, Hodson & Noble, 2014) and the Impact Map (Nicholls et al, 2012), among others. While substantively similar, each model represents a particular perspective on how to model impact (Ruff, 2021b). The similarities mean that it is possible to articulate a common model that can represent all these impact models thereby allowing the benefits of commonality without the draw backs of imposing a particular view.
Recently, The Impact Management Project harvested a consensus position of over 2000 practitioners to define Impact Dimensions. Figure 1 depicts the five types of information about change that are 4 http://ontology.eil.utoronto.ca/cids/cids.owl 5 https://impactmanagementproject.com/ needed for a robust understanding of impact plus a sixth dimension, how: An impact model must state what outcome has occurred in the period, whether the outcome positive or negative, and how important is the outcome to the people; It is also necessary to know who experiences the outcome, and how underserved are the affected stakeholders with respect to the outcome; how much change (duration, depth, duration) has been created; how much of that change is a result of the reporting organization’s activities; did the how contribute to the outcome or would it have likely happened anyways; and what risks are associated with the change if it does not occur as expected. (The Impact Management Project is for use by all types of organizations, not just social purpose organizations. It does not include the ‘how’).
The Common Impact Data Standard (CIDS) is designed to model a variety of impact models (how) and the five dimensions of impact (what, who, how much, contribution and risk). Figure 2 depicts many of the core classes and object properties in the ontology. The classes in yellow are used to define an organization’s impact model, and the classes in white are used to report on the impact the implemented model has on its stakeholders.
hasContributingStakeholder
Organization hasModel hasProgram hasService
Program Service
In the remainder of this section we briefly describe a subset of the core ontology patterns. For most patterns, core classes and object properties are depicted as a graph. Class Description Logic definitions, plus additional object properties and data properties, can be found in the CIDS specification and accompanying OWL file. The example used in the section is an SPO named “SfH” that provides skills training to homeless youth in Toronto.
The following prefixes are used for ontology namespaces: Prefix URI act http://ontology.eil.utoronto.ca/tove/activity# cids http://schema.org/ https://www.w3.org/2006/time# http://www.w3.org/2001/XMLSchema# 3.1.
A Social Purpose Organization (SPO) is represented as an instance of the Organization Class. An Organization can have one or more Impact Models (Figure 2) whose subclasses include Logic model, Logical Framework Analysis, Theory of Change, Outcome Chain, Impact Map and Outcomes Map. The properties of an ImpactModel are defined by the subclass. For example (Figure 3), an ImpactMeasurement model is defined has having Stakeholders, Outcomes, StakeholderOutcomes, ImpactReports, Indicators, and IndicatorReports. In our example, SfH would be an instance of Organization linked to an instance of ImpactMeasurement. Note that the “how” dimension of Program, Service and Activity is not part of this impact model, though it is part of a logic model. This allows the five dimensions (what, who, how much, contribution and risk) to be combined with any “how”. The remainder of the example classes are defined in subsequent subsections.
ImpactModel
An important component of an impact model is the Outcome it was designed to deliver. The “what” dimension specifies the outcomes for a selected set of beneficiary stakeholders. It can be used prospectively for planning, or retrospectively for reporting. It supports the answering of the following competency questions (we have expressed these as retrospective reporting, but each could be rephrased as prospective planning questions): 1. What outcome did occur in the period? 2. Is the outcome positive or negative? 3. How important is the outcome to the people (or planet) experiencing it?
While SPOs’ outcomes are necessarily highly specific to their work and context, the work is undertaken with an eye to bigger and broader goals. For example, SfH might work toward the outcome “homeless youth become qualified to work in construction industry”, which they measures as “number of youth (ages 14-25), who were rough sleeping or couch surfing in the preceding 3 months, in [catchment area], who attained machine operator certificate” This is a component of their funder’s outcome “increased employability”, which they measure as “number of people in the province with increased employment skills”. It is also part of a pathway, represented by the canProduce property, to broader goals such as decreased poverty and increased well-being. These broader goals can be identified by domain specific standards for both outcomes. For example, the United Nations Sustainable Development Goals (UNSDGs) Goal 1 is to “End poverty in all its forms everywhere”. CIDS can represent both custom and domain specific indicators and illustrates both components and pathways to allow an analyst to “roll up” from the specific to the more general. Figure 4 depicts the Outcome Pattern. An Outcome defines the intended impact the SPO will have on stakeholders. SfH’s identifies that its outcome is a component of with UNSDG outcome “8.6 By 2020, substantially reduce the proportion of youth not in employment, education or training”. Consequently, SfH’s outcome is linked via definingOrganization to the UN organization, and to the UNSDG outcome 8.6 via hasSpecification – both properties are inherited from Code. The characteristics of the beneficiary stakeholders are defined in the StakeholderOutcome class. It specifies several properties. First, who the outcome is for (hasStakeholder), i.e., youth or youth aged 14-25. Second, how important the outcome is for the stakeholder (e.g., “high importance”, “moderate important”, “neutral”, “unimportant”), from whose perspective it is important (fromPerspectiveOf). This property is included because increasingly SPOs are being encouraged to focus budget and reporting on the outcomes that are most important to those who are affected, rather than those that are most important to funders. Third, if the intended stakeholder is underserved. This property is used to help distinguish between social-purpose and not social-purpose businesses6. Fouth, the intended impact. This sets a target against which results can be compared. As StakeholderOutcome is specialized to a specific Stakeholder, a more specific outcome and set of indicators can be defined and used to measure the impact (hasIndicator), and the report of actual impact (hasImpactReport). 3.3.
The beneficiary of an outcome is a Stakeholder. A Stakeholder is a person or organization. It can be either a beneficiary of an SPO’s services or a contributor, e.g., goods, funding, expertise. There is a myriad of characteristics that SPOs might use to identify the beneficiary stakeholder. There are characteristics that the stakeholder must have (i.e., requirements) to be eligible for their services. There are also characteristics that the organization might track to learn more about which people are accessing the product or service. Common stakeholder characteristics are gender, age, race, income, geographic location, and disability. The specification of stakeholder characteristics is often domain dependent. For example, in the homeless domain, characteristics such as length (of time) and frequency of homelessness, and location of homelessness (e.g., street, shelter, friends home) are used to determine which services are relevant. Secondly, there often does not exist a single set of characteristics for a domain, but instead alternative sets of characteristics specific to an NGO, government organization, funder, etc. Competency questions include: • Who experienced the outcome? • Are all intended stakeholders benefiting? And benefitting equitably? • How underserved are the affected stakeholders in the relation to the outcome?
Simply listing a set of properties, the approach taken by vocabularies such as FOAF7 and Schema.org8, is insufficient for a number of reasons: 1. The possible set of properties associated with a Stakeholder is enormous when taking the union across domains and sources, leading to an overloading of the concept. 2. The plethora of properties across domains leads to ambiguous and overlapping interpretations. 3. The temporal aspect of a property is ignored, i.e., over what period of time is the property valid (see Katsumi & Fox (2019) for one possible approach to dealing with this).
4. The causal aspect of a property is ignored, i.e., what led to the Person having the property. Figure 5 depicts a graph of the Stakeholder pattern.
Stakeholder is a subclass of Person or Organization. The most important information about a stakeholder is their characteristics. In our example where the stakeholder is a youth, hasCharacteristic 6 The Impact Management Project refers to this as “C-Class” where C is for Contributing to solutions. More information available at https://impactmanagementproject.com/investor-impact-matrix/ 7 http://xmlns.com/foaf/spec/ 8 https://schema.org/ would link to an instance of Characteristic that would specify the age range of the person. Other characteristics, such as type of homelessness, whether indigenous, etc., would also be specified by separate instances of Characteristic. Characteristic extends the Code class enabling the reuse of characteristics defined by other organizations. 3.4.
The How Much and Contribution dimensions address the competency questions: • How much of the outcome is occurring – across scale, depth and duration? • Would this change likely have happened anyway? How Much measures the degree of impact an SPO has on its stakeholders. Contribution compares the degree of impact against a baseline, represented as a Counterfactual. While the previous dimensions are used to both define an SPO’s impact model and reported impact, these two dimensions are used only to report on the results of applying the impact model to stakeholders. ImpactReport reports on the effects on stakeholders experiencing a service provided by an SPO. In this case, it would report impact scale, depth and duration separately, where each includes the properties: hasIndicator: links to the Indicator used to measure the impact; prov:wasGeneratedBy: links to the activity that generated the impact value if an indicator is not specified; and hasCounterfactual: links to a Counterfactual which can be used to calculate what the impact on stakeholders would be if the stakeholders did not receive the service. The Counterfactual class specifies the spatial area over which the counterfactual was measured, the time interval, method of measurement and the value of the measurement.
In order to measure how much, a SPO specifies Indicators for each Outcome and StakeholderOutcome, that measure overall impact and stakeholder specific impact, respectively. The indicator for our example is “average number of skills each homeless youth attains.” CIDS provides an indicator pattern for defining indicators based on the Global City Indicator Foundation Ontology (Fox, 2013; 2015)9, which has been published as ISO/IEC technical standard 21972:2020. Figure 7 depicts a difference indicator pattern which is defined as having a taking the difference of two terms, each measuring a statistic (in the case size) of a Population. Membership in a population is defined by a class (Fox, 2018). In this example, the indicator takes the difference between the mean number of skills homeless youth have before and after participating in the SfH’s service.
i72: Mean i72: Population
Person hasSkills: hasAge:
i72: Indicator
i72:
Difference_Quantity Average number of skills each homeless
youth gained i72:term_1
i72:term_2 Average number of skills each homeless youth post intervention
i72:mean_of Homeless youth Group - post intervention
i72:defined_by i72:parameter_of_var
i72:mean_of hasSkills
Average number of skills each homeless youth pre intervention Homeless youth Group - pre intervention i72:defined_by i72: Mean i72: Population
Person hasSkills: hasAge: Another component of Outcome is a set of Impact Risks. ImpactRisk “assesses the likelihood that impact will be different than expected, and that the difference will be material from the perspective of people or the planet who experience impact.” ImpactRisk has nine subclasses that identify different types of risk10 each identified as “lowRisk”, “mediumRisk”, “highRisk”; hasConsequence, which identifies the degree of impact the risk could have on the stakeholders and hasMitigation, which is string that specifies a mitigation plan or references a document. This risk pattern is useful when interpreting data. It provides insights into why reported impact (the depth, duration and scale) might be less than expected. It provides insights when comparing the impacts of two SPOs by identifying if the outcomes are similarly risky. 3.6.
The program/service pattern, depicted in Figure 2, is motivated by the Canadian Government Reference Model (CGRM) (Wiseman, 2015): A program defines a set of services that focus on a shared set of Outcomes. For example, a “poverty reduction program” can be made up of a set of Services such as mobiles services that provides food and clothing to those that live on the street, and a training service that provides basic skills for those living on the street. A Program has the following object properties: hasService which identifies the Services that make up the Program; hasOutcome which identifies the Outcomes that the program is to achieve; hasContributingStakeholder which identifies the stakeholders that contribute to the Program; hasBeneficialStakeholder which identifies the stakeholders that benefit from the Program; hasInput which identifies the Inputs to the Program; and hasOutput: which identifies the Outputs of the Program. 9 http://ontology.eil.utoronto.ca/GCI/Foundation/GCI-Foundation-v2.owl 10 See: https://impactmanagementproject.com/impact-management/impact-management-norms/risk/
A Program has one or more Services which specify the activities that deliver the service. A Service has the following properties: act:hasSubActivity which identifies the Activities that make that comprise the Service; hasInput which identifies the Inputs to the Service; hasOutput which identifies the Outputs of the Service; hasOutcome which identifies the Outcomes that are specific to the Service; hasContributingStakeholder which identifies the stakeholders that contribute to the Service; hasBeneficialStakeholder which identifies the stakeholders that benefit from the Service; beneficiarySizeStart: which identifies the number of beneficial stakeholders at the beginning of the service time interval; beneficiarySizeEnd which identifies the number of beneficial stakeholders at the end of the service time interval; and i72:for_time_interval which is the time interval over which the service is provided. The representation of Activities is based on the TOVE Enterprise Ontology (Fox, 1992; Fox & Gruninger, 1998), including its Activity Ontology (Fox, Chionglo & Fadel, 1993), revised in (Katsumi & Fox, 2017). This information is important for questions of replication and learning what works. Some SPOs may wish to replicate the work of high-impact programs. This pattern explains “how” the impact was achieved. The pattern can also be used by managers of SPOs to inform innovations and trials. The program can be implemented differently in different geographies, time periods or for different stakeholders. A SPO manager can learn how to achieve greater impact by linking different Program, Service and Activity patters with the associated outcomes.
The Common Impact Data Standard has been evaluated through expert consultation and two stages of validation.
Expert consultations were initiated at the outset of the project and remain ongoing. At key points in development subject area experts were invited to participate in weekly or monthly meetings to advise the development of a particular portion of the Common Impact Data Standard. In total, five subject area experts were involved. They have contributed expertise in impact measurement standards, indicator standards, charity evaluation, and impact measurement software. In addition, five presentations to Common Approach partners solicited feedback from potential users including grantmakers, impact investors, software providers, impact measurement consultants, and SPOs. Finally, two public events brought forward input from broad range of subject area experts. There was a public webinar (70 attended) hosted by the Common Approach and a presentation at Good Tech Fest (May 2020). The indepth work with experts produced many revisions to CIDS v1 and v1.1. The user feedback and broader expert consultations reinforced both need and usefulness of such a standard, and the importance of using impact measurement software as an interface between the Common Impact Data Standard and those who that might use it. The ontology is too technical for the SPOs and their funders.
CIDS v1.1 has been through two stages of validation. The first stage sought to validate the flexibility of CIDS v1.1. We tested if CIDS v1.1 could be used to represent different accounts of impact. The research team built a sample of SPOs from our prior research for which we had impact data. From this sample, we selected one SPO that used a Logic Model and one that used an Outcomes Chain. We then represented each Impact Model using CIDS v1.1. The test demonstrates that CIDS V1.1 successfully represents both the Logic Model (Figure 8) and the Outcome Chain (Figure 9). These figures depict only a portion of the graphs.
The second stage sought to validate the completeness of CIDS v1.1. We tested if CIDS v1.1 could represent the entirety of an SPO impact report. From May-July 2021, we conducted a qualitative content analysis on the impact reports of 8 SPOs. (full details of method forthcoming). Each impact report was imported in NVivo coded in three broad categories. The code “CIDSV1.1” (which included a subcode for each class and property) as used to identify impact data in the report that could be represented by CIDS V1.1. The code “CIDS2” was used to identify impact data in the report that could not be represented by CIDS V1.1. The Code “Other” was used to identify report content that did not pertain to impact (subcodes include, repetition, heading, navigation, supplemental). A research assistant coded every word in each report. Results of the preliminary analysis show that CIDS V1.1 is able to represent most all of the impact of these organization (725 instances), however there are impact areas not supported by CIDS V1.1 that will be added to CIDS2. These include a representation of the need the SPO addresses (21 instances) and examples of impact, such as testimonials and stories that illustrate an outcome (50 instances).
Input “Clients who experience homelessness”
hasOutome We call this alignment. There are three levels of alignment: Basic, Essential and Full. A detailed statement of the alignment criteria are available online. Three software vendors have aligned or have signed a document committing them to align.
• Sametrica11 has committed to a timeline align at the full tier. Sametrica has implemented • • most of the classes and properties. properties are already implemented.
RIDDL12 has committed to a timeline to align at the Essential tier. Many classes and Helpseeker13 is incorporating CIDS as part of its Compass social services sector platform data model. The Compass project is funded by the Canada’s Digital Technology
The Common Impact Data Standard is also be used by social purpose organization, grantmakers and investors. We call this adoption14. One way that SPOs adopt is by using an aligned software. There are over 200 SPOs using the data standard through the aligned software. Another way that SPOs adopt is by creating their own databases using the properties and classes in CIDS. For example, an impact investor has used The Common Impact Data Standard to build a knowledge graph of their portfolio of investments. The knowledge graph, supported by the Common Impact Data Standard, allows a representation of the impact investor’s impact portfolio that would not be possible15 using traditional impact reporting practices (pdfs and excel sheets).
Just as the development of the SCOR reference model for Supply Chain Management (Stewart, 1997) has been transformational to the supply chain industry, the development of an ontology for impact modelling has the potential to be transformational to the Environment, Social and Governance sectors. The nature in which it is transformational is manifold: 1. It provides structure for how to think about modeling and measuring impact, e.g., making explicit both the expected outcomes, risks and how their achievement is to be measured, an area that has been historically qualitative; 2. It provides precise definitions of terminology thereby reducing the ambiguity of interpretation, and supporting the emergence of domain specific standards; 3. It fosters interoperability, i.e., the ability to understand and merge information available from datasets spread across social purpose organizations, their networks and their investors and grantmakers; 4. It makes possible the benchmarking of SPO performance, thereby making it possible to identify best practices and for SPOs to learn from each other; 5. It makes it possible for grantmakers to aggregate data across portfolios of investments; and 6. It makes the components of impact interpretable by a computer so that open source software and other technologies developed for big data can be applied to analyze and interpret the data collected and generated by social purpose organizations, including automating the detection of inconsistencies in data, as well as the causes of the observed variations.
Work continues on the development of the next version of CIDS, in particular an ontology of stakeholder needs. Secondly a repository is being developed to support sector wide analysis.
This research was supported, in part, by the and the Ministry of Employment and Social Development Canada, the Ontario Ministry of Economic Development & Growth, and the Natural Science and Engineering Research Council Canada. We wish to thank the contributions made by Tawfiq Abdulai, Anshula Chowdhury, Bart Gajderowicz, James Hicks, Daniela Rosu and Jane Zhang to the design of CIDS.
Allen, W.H. (1906), “Hospital Efficiency”, American Journal of Sociology, Vol. 12 No. 3, pp. 298– 318. 14 https://www.commonapproach.org/common-impact-data-standard/ 15 https://www.commonapproach.org/wp-content/uploads/2021/03/Common-Impact-Data-Standard-V1.1-Alignment-Tiers-by-class-andproperty.pdf Earl, S., Carden, F. and Smutylo, Q. (2001), “Outcome mapping: building learning and reflection into development programs”, Ottawa: International Development Research Centre, www.betterevaluation.org/resource/outcome_mapping (accessed 16/09/2020) Ebrahim, A. and Rangan, V.K. (2014), “What impact? A framework for measuring the scale and scope of social performance”, California Management Review, Vol. 56 No. 3, pp.118–141. Harries, E., Hodgson, L., & Noble, J. (2014). Creating your Theory of Change: NPC’s practical guide, www.thinknpc.org/resource-hub/creating-your-theory-of-change-npcs-practical-guide/ (accessed 06/30/2021).
Fox, M.S., (1992), “The TOVE Project: A Common-sense Model of the Enterprise”, In Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, Belli, F. and Radermacher, F.J. (Eds.), Lecture Notes in Artificial Intelligence # 604, Berlin: Springer-Verlag, pp. 25-34. Fox, M.S., (2013), “A Foundation Ontology for Global City Indicators”, Working Paper, Enterprise
Integration Laboratory, University of Toronto, Revised: 13 October 2017.
Fox, M.S., (2015), “The Role of Ontologies in Publishing and Analyzing City Indicators”, Computers,
Environment and Urban Systems, Vol. 54, pp. 266-279.
Fox, M.S., (2018), “Semantics of Populations: A City Indicator Perspective”, Journal of Web
Semantics, Vol. 48, pp. 48-65.
Fox, M., Chionglo, J.F., and Fadel, F.G., (1993), “A Common Sense Model of the Enterprise”, Proceedings of the 2nd Industrial Engineering Research Conference, pp. 425-429, Norcross GA: Institute for Industrial Engineers.
Fox, M.S., and Gruninger, M., (1998), “Enterprise Modeling”, AI Magazine, Fall 1998, pp. 109-121. Fox, M., Ruff, K., Chowdhury, A., Gajderowicz, B., Abdulai, T., Zhang, J., (2020), "The Common Impact Data Standard: An Ontology for Representing Impact, Version 1.1", Common Approach Project Report. https://www.commonapproach.org/wp-content/uploads/2021/03/Common-Impact-Data-StandardV1.1.pdf Katsumi, M. and Fox, M.S., (2017), “Defining Activity Specifications in OWL”, Proceedings of the 8th Workshop on Ontology Design and Patterns, Vienna.
http://ontologydesignpatterns.org/wiki/WOP:2017#WOP_2017_Program Kendall, J., & Knapp, M. (2000). Measuring the performance of voluntary organizations. Public
Management Review, 2(1), 105-132.
Nicholls, J., Lawlor, E., Neitzert, E, and Goodspeed, T. (2012), “A guide to social return on investment”. http://www.socialvalueuk.org/resources/sroi-guide (accessed 24/02/2019). Oakes, L.S. and Young, J.J. (2008), “Accountability re-examined: Evidence from Hull House”,
Accounting, Auditing and Accountability Journal, Vol. 21 No. 6, pp. 765–790.
Paton, R. (2003). Managing and measuring social enterprises. Sage.
Practical Concepts Incorporated (1979), The Logical Framework: A Manager’s Guide to a Scientific Approach to Design & Evaluation, www.eval.fr/wpcontent/uploads/2018/03/the-logicalframework-eval.fr_.pdf (accessed 08/10/2020).
Ruff, K (2021a) “Performance Measurement” in Intersections and Innovations: Change for Canada’s Voluntary and Nonprofit Sector Edited by Susan Phillips and Bob Wyatt, www.muttart.org/wpcontent/uploads/2021/05/Chapter-33-Social-and-Environmental-Impact.pdf Ruff, K. (2021b), "How impact measurement devices act: the performativity of theory of change, SROI and dashboards", Qualitative Research in Accounting & Management, Vol. ahead-ofprint No. ahead-of-print. https://doi.org/10.1108/QRAM-02-2019-0041 Ruff, K & Olsen, S., (2016), “The next frontier in impact measurement isn’t measurement at all: why we need skilled analysts to improve social capital markets”, Stanford Social Innovation Review. May 10, 2016.
Stewart, G. (1997). Supply-chain operations reference model (SCOR): the first cross-industry framework for integrated supply-chain management. Logistics information management. Weiss, C.H. (1997), “Theory-based evaluation: Past, present, and future”, New Directions for
Evaluation,Vol. 1997 No. 76, pp.41–55.
Wiseman, R. (2015). Canadian Governments Reference Models. In Service Systems Science (pp. 109128). Springer, Tokyo.