Effective social services require robust program design and rigorous evaluation, yet many initiatives still lack mechanisms for formalizing and operationalizing stakeholder needs, goals, and outcomes. This often leads to inefficient delivery and inconsistent assessment. This paper reports on a preliminary, qualitative study assessing the usability of our representational framework designed to operationalize stakeholder goals, needs, and outcomes within the social services sector. Results indicate that the framework significantly enhanced the ability of program designers and evaluators to define target beneficiary characteristics, design indicators for measuring changes induced by programs in the target characteristics and formulate intended program outcomes, leading to more coherent program design, aligned with stakeholder needs and priorities, facilitating data-driven, outcome-based evaluation and improving communication across all stakeholders
Effective social services require both appropriate design and rigorous evaluation to ensure they meet the needs of their clients and achieve the intended outcomes for all stakeholders involved, from beneficiaries and service providers to policymakers and communities at large. Social programs present unique challenges as they have to balance the interests of multiple stakeholder groups, from beneficiaries and service providers to policymakers, operate in dynamic environments where social, economic, and political factors can significantly influence program effectiveness and also account for possible implementation variability, as the actual operation may differ from the program design in ways that can substantially affect results.
Program design must be strategic and the execution continuously monitored in order to
achieve long lasting positive impact. This process often starts with conducting an assessment
to identify the needs of the target populations and current service gaps and continues with
1Proceedings of the Joint Ontology Workshops (JOWO) - Episode XI: The Sicilian Summer under the Etna, co-located
with the 15th International Conference on Formal Ontology in Information Systems (FOIS 2025), September 8-9, 2025,
Catania, Italy
*Corresponding author.
† These authors contributed equally.
defining intended outcomes (i.e., changes in client knowledge, skills, behavior, etc.) and
establishing measurable objectives to ensure alignment with stakeholder priorities. Providers
of social services also often develop a theory of change, or logic model to help map out how
program activities will lead to the desired outcomes, in many cases using a participatory model
which extensively involves stakeholders from early on in the process to increases commitment
to and improve the chances of success. Research suggests that embedding structured
methodologies into program planning also allows for the creation of more precise evaluation
models that better align with the realities of service delivery [
Formalizing stakeholder needs, goals, and outcomes can help in multiple ways, not in the
least by enhancing precision, as clearly defined objectives allow for better measurements and
comparison across programs [
We report here on our work to investigate the applicability of the framework we developed for formalizing needs, objectives, and outcomes to improving both the design and assessment of social services, in particular social programs for youth [12, 13, 14, 15]. We conducted this investigation with the help of our collaborators at a Toronto-based agency that offers a large array of programs for newcomers of all ages, focusing on programs for newcomer youths.
Our study was focused on two main research questions 1. 2.
Does using our framework support more efficient design and implementation (if yes, in what way)?
Does our framework support better program evaluation (if yes, in what ways)? The paper is organized as follows: Section 2 provides a brief background, Section 3 outlines the methodology used, Section 4 presents the findings, and the final section discusses implications and future research directions.
Canadian settlement agencies are funded by the federal government, as well by provincial and municipal agencies to provide services such as language training, employment assistance, housing support, and cultural orientation to immigrants and refugees. These agencies assist newcomers with accessing healthcare, navigating administrative and legal processes and connecting with local communities, fostering social inclusion and playing a crucial role in helping them integrate into the Canadian society.
Specialized services are geared specifically towards helping newcomer students and their parents integrate into the Canadian school system. A variety of programs, such as the widely implemented Settlement Workers in Schools, provide school orientation, academic guidance, and referrals to community resources, helping students navigate the education system while also fostering a sense of belonging and assisting families in identifying barriers to integration in a culturally sensitive environment. They also offer linguistic and cultural interpretation to improve communication between students, parents, and educators as well as mentoring and tutoring programs help students stay on track academically while they adjust to the new learning environment.
Ready for School is one such initiative. It is a two-week summer program run by JIAS Toronto, a settlement agency, and introduces newcomer children and their families to the Ontario school system, familiarizing them with daily routines and the Canadian school environment. Ready for School is designed for children in grades 4 to 12 and their parents and provides an opportunity for participants to experience a school-like setting and socialize with peers before the academic year begins, as well as practice their language skills, introducing them to essential English terms related to school and beyond. Due to its short duration and annual recurrence, Ready for School presented an excellent opportunity to test how effectively the framework we developed over the past few years for representing stakeholder goals, needs, and outcomes can support program design, monitoring and evaluation.
The traditional logic / change models used to design and evaluate social services programs and initiatives typically use a standardized representation that groups relevant information into categories, e.g., inputs, activities and (intended) outcomes and visually organize the categories as separate column tables, without offering mechanisms to explicitly assign inputs and outcomes to activities, describe the internal structure of complex activities or sequence activities based on temporal dependencies. They also commonly lack mechanisms for associating outcomes with the stakeholder characteristics they are related to and with the indicators designed to help assess whether those outcomes have been achieved. Although easy to read due to their familiar tabular formatting, these logic models are unable to provide support for establishing clear means for continuously monitoring the implementation of programs, to identify the contribution of each activity towards the achievement of the intended outcomes and to discover and remedy bottlenecks and areas of underperformance.
Over the past few years, we have developed a representational framework designed to help operationalize the goals, needs, and outcomes of stakeholders within the social services sector. This framework provides a formal approach to understanding and aligning the design, delivery and evaluation of social programs with stakeholder expectations, preferences and priorities. For a more in-depth discussion, we encourage the reader to explore additional references [12, 13].
Establishing a clearer connection between the individual components of a program not only enhances their effective implementation but also provides a foundation for assessing alignment with stakeholder goals and intended outcomes through evidence-based, deductive reasoning. Beyond facilitating human-led evaluation, formal (structured) representation of client characteristics and needs also opens the possibility to use automated reasoning tools, allowing computational systems to assess conceptual coherence and alignment with stated objectives. This capability can support ongoing, real-time measurement of outcome achievement, ensuring that the effectiveness of the program can be continuously monitored. Additionally, automated reasoning can identify potential gaps in the original program design and suggest activities for inclusion, helping program designers refine and optimize social interventions.
This study serves as an initial (observational) assessment of the framework’s usability, examining its practical application and potential for improving program planning, monitoring, and evaluation processes. It functions as a proof of concept to lay the groundwork for broader application and more in-depth evaluation. Given the scale of the program involved in the study, the assessment methods employed were qualitative, largely based on direct observation and input from program designers, managers, and evaluators.
Our study centered on two key research questions aimed at assessing the potential impact of our framework in the context of social services programming and delivery. First, we examined whether applying the framework contributes to a more efficient program design and implementation. This involved investigating how the representation of stakeholder needs, goals, and outcomes proposed in our framework influenced planning processes, decisionmaking and resource allocation. Second, we studied whether our framework enhances program evaluation, specifically in terms of optimal choices of measurement criteria, indicators design, assessment accuracy, and ensuring that the proposed evaluation methodology aligns with the intended goals of the program
Our process involved training the program designers, managers and evaluators in the application of our representational framework. Four people directly responsible for the design, management and evaluation of the program were trained. Following this training, we observed how the framework's use impacted multiple dimensions of program development, management and evaluation. Specifically, we examined how the framework influenced the design of the overall program structure, the formulation of activities, their inputs and outputs and the articulation of intended outcomes at program and activity level. We also observed its effects on the design and execution of the evaluation methodology by the dedicated evaluation professional engaged by the collaborating NGO. Additionally, we examined how the application of the framework influenced communication with beneficiaries and other stakeholders at various stages of the program. This included assessing its influence on the refinement of the program description, specifically around the communication of the program's available services, objectives and intended outcomes, as well as analyzing its impact on the messaging provided to parents of the enrolled children, particularly with regards to tracking their children's progress and maintaining transparency throughout the program. We also investigated how the framework contributed to the articulation of the findings of the professional evaluator regarding the program's effectiveness, areas for improvement, and participant outcomes.
Ready for School is a two-week summer program that introduces newcomer children in grades 4 to 12 and their families to the Ontario school system. Its original stated overall objective is “To orient newly arrived school aged children to the Canadian school norms and expectations, expand their social circles, and improve their English language skills, which in turn will promote an easier adjustment to school.” Through various group activities and workshops run by volunteer teachers, but outside of the formal schooling system, it supports integration in the new educational environment by assisting with practicing English language skills and life skills development, promoting social-emotional learning and helping ensure that newcomer students and their parents feel welcomed and empowered as they transition into Canadian schools.
We began by providing comprehensive training to the professional program evaluators engaged by the NGO administering the initiative, equipping them with the knowledge and skills necessary to effectively apply the representational framework. We trained them on how to: 1. identify stakeholder characteristics, e.g., age, knowledge of the Canadian school norms, English speaking proficiency, attitude towards attending school, size of peer network/number of friends, strength of local social network, etc. 2. define actual and desired states for stakeholders by associating the selected characteristics with values from a set of possibilities, e.g., inadequate knowledge of the basic Canadian school norms, poor English speaking proficiency; 3. formulate possible stakeholder goals based on desirable states and timelines for achievement, such as mid-way through the program, at the end of the program, six months post-program, etc. (e.g., be able to effectively navigate the Canadian school environment by the end of the program, be able to communicate in English simple classroom-related requests by the end of the program, have a supportive peer network, by the end of the program; 4. state potential stakeholder needs based on the identified characteristics, e.g., need to improve knowledge of the basic Canadian school rules, need to improve English speaking proficiency, need to have more connections with people of the same age; 5. articulate intended outcomes, e.g., knowledge of Canadian school norms advanced to satisfactory, English speaking proficiency advanced to adequate, the number of friends increased by at least 1.
To understand why parents and children are seeking to enroll in the program, a questionnaire is provided at the time of application for them to communicate their goals and motivation. The program designers, as well as the evaluators have both access to the results of these questionnaires. We specifically focused on the impact on the evaluators' identification of the beneficiary characteristics that the program aims to impact, design of the indicators used for measuring changes in those characteristics and formulation of intended program outcome, specifically with regards to the impact of using the framework on linking the identified stakeholder characteristics to the expected program outcomes
We gathered data through a combination of direct observation and feedback from the evaluators during the program evaluation design process. In previous iterations of the program, the evaluations relied on notes from volunteer teachers and narrative descriptions from coordinators, lacking direct measurement of the client characteristics the program aimed to affect.
By observing the evaluators as they applied the representational framework during their work, we were able to assess how it influenced their approach to program assessment. For example, we noted that familiarity with our framework enabled the evaluators to identify and formalize the key client characteristics, such as “knowledge of Canadian classroom norms”, “attitude about attending school”, “ability to make meaningful relationships with others”, that could be directly assessed during program intake and then measured mid-way through the program as well as at the end in order to provide a base line and allow for progress mapping, improving on the processes used in the previous years the program was offered. Understanding the framework also helped them with defining the needs to be assessed, relevant indicators, and assessment criteria. We actively engaged with the evaluators throughout their work, soliciting their reflections on the framework’s utility, challenges encountered, and adjustments they would recommend based on their experiences. This dual approach allowed us to capture both objective insights from observed practices as well as subjective perspectives from evaluator feedback, providing us with a well-rounded understanding of the framework’s impact on the design of program evaluations. We then conducted introductory and training sessions with the program designers and managers, equipping them with the knowledge and tools necessary to use the representational framework in their work.
With rising competition for funding, social service organizations have to complement or replace the narrative reports that are standard in the field with data-driven, evidence-based ones. This makes the ability to define measurable outcomes and provide data to support their achievement crucial for success.
We observed, as the designers and managers engaged with the framework, how it influenced their approach to refining the existing program design, and theory of change, particularly in terms of being able to clearly distinguish between, as well as define outputs, e.g., (number of) English language essays written by the participants, (number of) group conversations held, (number of) opportunities to make community connections and intended outcomes, e.g., majority of participants improved their English speaking proficiency by 1 level, 100% of student participants know the basic Canadian classroom norms, the majority of student participants report making at least one friend and being able to link the program activities to the client the characteristics they were intended to affect.
We also examined how the framework influenced program managers in designing and executing communication strategies. Our primary focus was to determine if the formal representation of stakeholder goals, needs, and outcomes facilitates a clearer communication. In particular, we wanted to find out if it facilitated a clearer articulation of the program’s design and objectives to the parents of enrolled children and if it enabled a more effective communication to the external funders of the program's impact on participants and their immediate community, including the schools the children would attend. It is worth noting that the evaluation of the program continues beyond the immediate activities. This is accomplished by the program evaluators via liaising with the settlement social workers posted in the schools attended by former program participants
Traditional social program design and evaluation methods often lack a structured approach to
formalizing stakeholder needs, goals, and expected outcomes and group program components
into vertical categories such as “inputs”, “activities”, etc. obscuring or ignoring the links
between individual inputs, activities, outputs and outcomes. Although the evaluation of social
services programs is crucial to ensuring that interventions align with stakeholder needs and
achieve desired outcomes, evaluators, who are well-versed in classical social science
methodologies [
Formalizing stakeholder needs, goals, activities and outcomes is a necessary step forward in the design, implementation and evaluation of social services. It has the potential to not only enhance stakeholder alignment and accountability, but also to lead to improved impact as programs become evaluation-ready from inception and potential issues can be identified and corrected while the program is ongoing. Formal representational frameworks provide an effective approach to achieve these improvements [12, 13, 14].
Identifying and formalizing the client characteristics that social program activities are designed to impact, together with formalizing the relationships between these characteristics, can help surface the links between program activities and facilitate a more integrated program design, ensuring that interventions align with client goals and needs and that program evaluations can move beyond narrative descriptions of impact and be based on data-driven processes.
Our framework for formalizing characteristics, goals, needs, activities and outcomes [12, 13] allows for systematic measurements collected at the individual client level, generating data that can truly reflect the effect of different program activities, providing accurate insights into which aspects of the program are driving change, which strategies are effective, and where adjustments may be needed.
Following the training sessions, we observed how the implementation of the framework influenced various aspects of the program design and evaluation processes. A key focus was observing the framework’s impact on how the evaluators and the program designers identified beneficiary characteristics that the program aimed to influence, as formalizing these characteristics allows for a more precise definition of the target changes the program aims to enact.
During the observational phase of our study we noted how the professional evaluators applied the framework to identify new beneficiary characteristics, e.g., knowledge of the Canadian school norms, define client states to be included in a new intake/need assessment form for the program, e.g., knowledge of the Canadian school norm is poor (client's current state), have adequate knowledge of the Canadian school norms by the end of the program (client goal, expressed as desired state to be achieved in a particular time frame), as well as refine or design new activity outputs, activity outcomes, e.g., improved knowledge of the Canadian school norms, and program outcomes, e.g., 100% of student participants have increased their first-hand experience of Ontario classroom by 10 hours.
Fig. 1 presents an excerpt from the new logic model for the Summer 2024 edition of the Ready for School program, designed by the professional evaluators after being trained in how to use the framework.
After analyzing how the framework shaped the formulation of beneficiary needs and intended program outcomes we noted that the revised program intake form exclusively incorporates measurable stakeholder characteristics, such as “first-hand experience of Ontario classroom” (quantified in classroom-related hours), “number of friends”, (measured as a count), “knowledge of Canadian school norms and expectations” (measured on a Likert scale), knowledge of Canadian history and geography (evaluated using a Likert scale) and that the evaluators clearly stated the connection between stakeholder characteristics, needs and expected results, using the framework. This is demonstrated by the composition of the logic model, as well as by the remainder of the evaluation methodology designed and implemented for the Ready for School program. (A full description of the logic model and the evaluation process will be presented at a professional conference later this year. Training materials are in preparation and will be shared with the social services sector prior to the conference.) The approach evaluated here also helps ensure that the goals and needs of participants are directly linked to measurables and facilitates data-driven program monitoring and evaluation.
By supporting the articulation of clear connections between identified needs and impact, the framework enhanced the ability of the evaluators to define measurable indicators to and test the alignment between the implementation of the program and its design and stakeholder intentions, improving data collection and providing a robust foundation for assessing the success of the initiative, increasing the likelihood of its continued external funding. Some of the indicators developed, e.g., number of friends, strength of social network, verbal English language skills, have been selected for implementation in subsequent extensions of the organizational information system (used for storing client profiles, goals, needs and outcomes).
We found that adopting the framework-based approach facilitated the planning of observations and measurements at the individual client level, enabling continuous program monitoring and iterative refinements. This shift empowered program managers to develop a new data collection strategy, supported by extensions to the existing organizational information system. By collecting individual-level information, specifically, the measurable characteristics identified by program designers and evaluators, program outcomes could be determined through the aggregation of this data. This methodological shift allowed program managers to go beyond subjective narrative descriptions of impact to truly evidence-based social impact measurement.
We received as feedback the recommendation that the framework needs to be extended with functionality for describing and checking, among others, temporal limits for achieving outcomes (e.g., six months since the start of a program, or the registration of a client goal) and numerical thresholds for describing improvements with respect to a recorded base line.
During this task-based evaluation pilot, data was collected and managed manually and the calculations / inferences necessary to ascertain the achievement of the intended outcomes were also performed manually. However, the framework supports inclusion in an information system and the automatic inference of need satisfaction and outcome achievement
Effective social services require robust program design and rigorous evaluation, yet many initiatives still lack effective mechanisms for capturing stakeholder needs, goals, and outcomes. This often leads to inefficient program delivery and inconsistent and incorrect impact assessment.
We report in this paper on a preliminary, task-based, qualitative evaluation assessing the usability of our representational framework [12] for operationalizing stakeholder goals, needs, and outcomes within the social services sector. Serving as a proof of concept, this investigation was conducted in collaboration with a Toronto-based settlement agency, focusing on their Summer program for newcomer youth as our pilot framework implementation.
Our study involved training the program's designers and professional evaluators in the framework's application. We then performed a task-based evaluation, observing the framework's impact on program design, on the evaluation methodology formulated for the program and on the strategy used to communicate with beneficiaries and other stakeholders. Given the program's size, our assessment methods were qualitative, based on direct observation and feedback from program designers, managers, and evaluators.
We observed that clearly defining the beneficiary characteristics targeted by various program activities, as well as outlining the connections between these characteristics, enabled program designers and evaluators to better meld the activities that comprise a social initiative into conceptually sound unit and plan for its effective evaluation. It also improved communication and stakeholder engagement, facilitating clearer messaging about program objectives, expected outcomes, and delivery, ensuring that beneficiaries and funders better understand the purpose, expected impact, and alignment with their needs and priorities. Instead of general messaging about improvements in abilities described at a higher, abstract level, parents could receive tailored reports based on the actual measurements of the changes effected by the activities of the program in the targeted student characteristics, e.g., “improved strength of social connections by 15%”.
Embracing formal representation not only enhances program and service efficiency, but can also strengthens their potential to continuously adapt, ensuring relevance to evolving stakeholder needs.
We note that the effects we observed at program level extend beyond that, influencing the organization's broader operational framework by advancing its data maturity and reinforcing a culture of accountability. The integration of evidence-based evaluation practices, facilitated by the application of the framework from program/service design and evaluation to data collection and higher-level analytics, allows organizations to refine their strategic approaches, strengthen transparency, and ensure that their initiatives remain aligned with stakeholder expectations, including those of external funders. The insights made possible in this context can influence decision-making from the front-line workers to the highest levels of organizational governance.
In conclusion, our study found that applying the framework enhances both program development and the monitoring and evaluation processes, leading to more coherent program design and improved communication across all stakeholders, paving the way for more effective and accountable social interventions.
Building on these findings, we plan to expand the pilot project to include additional NGOs as well as funders of social services. This will allow us to assess the framework’s ability to support integrated evaluation of social initiatives and facilitate the implementation of competitive, outcome-based funding models.
This research was made possible through the invaluable participation and support of JIAS Toronto, with special thanks to Oded Oron and Malka Elkin for their contributions
During the preparation of this work the authors used Google Docs and MS Word and their built-in tools for grammar and spelling check. After using these tool(s)/service(s), the authors reviewed and edited the content as needed and take full responsibility for the publication’s content [12] D. Rosu, M. S. Fox, B. Gajderowicz, Compass Needs Ontology: A Design Pattern for Representing Needs in Social Work, Joint Ontology Workshops (JOWO’22), Jönköpin, Sweden, 2022 [13] M. S. Fox, B. Gajderowicz, D. Rosu, A. Turner, L. Lyu, An Ontological Approach to Analysing Social Service Provisioning, IEEE International Smart Cities Conference (ISC2), 2022. [14] B. Gajderowicz, D. Rosu, M. S. Fox, Compass Event, Client, and Service Ontology: A Design Pattern for Social Services, Joint Ontology Workshops (JOWO’22), Jönköpin, Sweden, 2022.