=Paper=
{{Paper
|id=Vol-2916/paper_22
|storemode=property
|title=Socio-technical co-Design for Accountable Autonomous Software
|pdfUrl=https://ceur-ws.org/Vol-2916/paper_22.pdf
|volume=Vol-2916
|authors=Ayan Banerjee,Imane Lamrani,Katina Michael,Diana Bowman,Sandeep Gupta
|dblpUrl=https://dblp.org/rec/conf/ijcai/BanerjeeLMBG21
}}
==Socio-technical co-Design for Accountable Autonomous Software==
https://ceur-ws.org/Vol-2916/paper_22.pdf
Socio-technical co-Design for accountable autonomous software∗
Ayan Banerjee, Imane Lamrani, Katina Michael, Diana Bowman, Sandeep K.S. Gupta
Arizona State University
{abanerj3,ilamrani, katina.michael, diana.bowman, sandeep.gupta}@asu.edu
Abstract
Recently, efforts to regulate software and make or-
ganizations and individuals more accountable for
its consequences have increased. Traditionally, the
human-in-the-loop (operator, user, or bystander) is
usually blamed for undesirable behavior of soft-
ware systems in the real world. This is due to
the limitations of the user-centered design approach
where an average user’s mental model (MM) is
adopted. The core belief in this paper is that
user-centered design must incorporate a wider lens
of stakeholder interactions using socio-technical
ecosystems being inclusive of users from various
backgrounds and consulting with certifiers, man-
ufacturers, and regulatory agencies for a given Figure 1: Socio-Technical ecosystem to address accountability in
jurisdiction. We envision a socio-technical co- user-centered software design for autonomous systems.
design approach for the development of compli-
ant autonomous socio-technical systems (ASTS),
which can infuse novel interpretations of regula- where manufacturers focus on the users and their needs en-
tions based on the social, behavioral, and economic gaging the participant to understand their interpretation of
(SBE) background of users. We posit that an ac- regulation into a set of requirements for the operational char-
countable software has three properties: 1) opera- acteristics of the ASTS software [Robertson et al., 2019].
tional transparency: Amenable to monitoring rele- Specific software modules are then developed and tested to
vant parameters for tacit knowledge (user’s MM of ensure compliance to the requirements.
the ASTS), 2) operational adaptability: The soft- The interpretation of regulation is a function of the social,
ware can be configured to support evaluation of reg- behavioral, and economical background of a user and can dif-
ulatory compliance with changing performance ex- fer significantly across population. However, the need for the
pectations and compliance perceptions, and 3) op- development, certification and marketing of a minimum vi-
erational interpretability: The software can assist able product (MVP) within time constraints often result in an
in generating feedback for guidance on the com- interpretation of regulation that restricts compliance to lim-
pliance properties of novel modes-of-operations – ited usage configurations in the ASTS software. As such
a consequence of dissonance between MM of the compliance may not hold when the social, cultural, behav-
user and the system designer’s view of users’ MM. ioral and economic (SBE) background of a user results in
novel interpretations of the regulation and non-certified us-
1 Introduction age configurations. In such cases, there currently exits no
clear pathway towards ascertaining accountability to regula-
Compliance-by-design is useful to ascertain accountability tion. The core belief in this endeavor is that user-centered
in software design for autonomous socio-technical systems design must incorporate a wider lens of stakeholder inter-
(ASTS) [Graafstra et al., 2010]. The key feature of compli- actions using socio-technical ecosystems being inclusive of
ance is in the incorporation of regulations in software imple- users from various backgrounds and consulting with certi-
mentation. Recent initiatives such as the General Data Pro- fiers, manufacturers, and government agencies.
tection Regulation (GDPR) compliant-by-design ASTS are We envision that operational safety can be assured with
a testimony to this effort [Truong et al., 2019; Winfield et a socio-technical co-design approach for the development of
al., 2019]. A user-centered design approach is undertaken, ASTS software that embeds regulatory compliance from the
∗
This project is partly funded by DARPA AMP project. outset. In this approach, the user is considered as an ex-
Copyright © 2021 for this paper by its authors. Use permitted under pert in the experience domain and actively contributes to the
Creative Commons License Attribution 4.0 International (CC BY accountability of the software design based on regulations,
4.0). laws, applicable standards and risk management. The experi-
Figure 2: State-of-Art user-centered approach towards accountable software design, Limited SBE involvement.
ence of a user is a function of the exposures [Robertson et al., device should not raise different questions of safety and ef-
2017] and the user’s mental model of the ASTS. The mental fectiveness than another legally marketed device.
model is directly affected by social (e.g., age, gender, educa- In the user-centered approach (Figure 2) the manufacturer
tion levels), cultural (e.g. customs, ethnicity, spritual beliefs, utilizes expert advice from secondary sources of evidence to
and practices), behavioral (e.g. perceived risks and benefits make a population average mental model that translates the
of adoption of ASTS) and economic background (e.g. afford- regulatory law into requirements on the operational charac-
ability and insurance coverage) and is tacit knowledge em- teristics of the software. For the example case, the 510K pre
bedded in the operational characteristics of a deployed ASTS market approval regulation is converted to two requirements:
software [Robertson et al., 2017]. a) effectiveness, percentage of time in normo-glycemic range
The key feature of the socio-technical ecosystem (Figure (TiR) greater than that reported in sensor augmented pump
1) is the introduction of a trans-disciplinary liaison as a stake- (SAP), and b) safety, low percentage of time in hypoglycemia
holder representing the user population in the user-centered than in SAP [Berget et al., 2020].
design approach. A liaison acts as intermediary interlocu- Through interaction and risk analysis the manufacturer
tor between the user population and the other stakeholders then builds software add-ons to the MVP to address regu-
who can: a) provide tacit knowledge to the other stakehold- lations. For the case of Minimed pump, this is a supervi-
ers about the effects of social, cultural, behavioral and eco- sory control software component to automatically deliver in-
nomic characteristics of user cohorts on interpretation of per- sulin between manually announced meals to address effec-
ception of compliance and performance expectations from an tiveness, and two safety modules, suspend on low and sus-
ASTS, and b) explain compliance and operational character- pend on predicted low. The manufacturer then collaborates
istics of the ASTS software to the user to ethically align their with domain experts to conduct in-the-field studies to collect
interactions with the ASTS software [Michael et al., 2021]. data on the compliance and performance properties. The data
Socio-technical co-design attempts to improve software ac- driven compliance argument is then submitted to the certifier
countability with respect to regulation using a three pronged (FDA in case of Minimed 670G) for regulatory approval. The
approach (Figure 1): state-of-art user-centered design approach has the following
a) Extraction of tacit knowledge: A trans-disciplinary drawbacks:
team (e.g. computer scientists, CSE and SBE scientists) at-
tempts to extract conceptualization of user’s mental model Limited reconciliation pathways can result in unmet ex-
and novel usage configurations of the software from continu- pectations: Population average models may not be appli-
ously monitored operational characteristics of the software. cable for a given user. In such scenarios, the performance
b) Certification game: Continuous evaluation of novel us- expectations may not be met. In the current state-of-art the
age configurations for regulatory compliance through a certi- pathway towards a reconciliation is vague and unlikely.
fication game between stake holders. For example, studies [Berget et al., 2020] show that many
c) Feedback to stake holders: Generating feedback for Medtronic 670G users (actors) experience significantly less
the actors and stake holders to reshape mental models and TiRs than that observed in clinical studies used for showing
improve perception of accountability. regulatory compliance [Berget et al., 2020]. A primary rea-
son for this is that several actors spend less than 60% of time
per day in auto mode, where the PID controller is active. The
2 Accountability issues in state-of-art auto mode exits are triggered by two sources: a) the safety
In the state-of-art user-centered approach (Figure 2) the man- modules suspend insulin delivery when glucose levels reach
ufacturer first develops a MVP to meet a technological and or are predicted to reach low values, and b) the auto mode
market demand gap. The required regulations are then care- defaults to safe basal mode nearly 15% of the time when glu-
fully evaluated by the manufacturer in collaboration with a cose levels are high for a long duration. The safe basal mode
law expert, taking into account the regulatory agency guid- injects a constant level of basal insulin and does not utilize
ance. For example, semi autonomous medical devices such the PID algorithm. Such usage artifacts were not observed in
as closed loop blood glucose control systems (Minimed 670G clinical studies which report 95% time in auto mode [Berget
by Medtronic) are classified as class II devices by Food and et al., 2020]. This is a violation of the effectiveness compo-
Drug Administration (FD& C Act) and are accountable to nent of regulation and currently there is no pathway towards a
510K pre-market approval. The regulatory law states that the reconciliation. In fact, this pushes the user towards initiatives
such as Do-It-Yourself (DIY) [Ahmed et al., 2020] insulin can express the control decisions as a set of differential equa-
pumps that allow unsatisfied users to design their own con- tions. The transitions between each mode is governed by
trol software. external events such as meal intake or internal events when
Unmet expectation may lead to ethically misaligned user blood glucose levels are within specific ranges.
behavior: Ethical values are a function of social and behav- Mental Model: It is the perceived operational characteristics
ioral background of an user. Unmet expectations from a soft- and performance expectations of a citizen user. The mental
ware can often trigger the user to subvert compliant operation model guides the interaction of the user with the exposures of
of the software. For example, a user can have a performance an autonomous system to achieve the expected performance
expectation to minimize post prandial glucose level instead of for a given environmental context. It also guides the inter-
increasing TiR. (Post prandial hyperglyemia has a strong pos- pretation of a regulation into requirements on the operational
itive correlation with HbA1C levels [Ferenci et al., 2015], the model. Hence, it is a connector between the user, certifier,
gold standard metric for Type 1 Diabetes.) In such a scenario, oversight, and manufacturer.
the safety module of suspend on low or predicted low may in- The mental model is influenced by the socio-cultural back-
duce unnecessary auto mode exits resulting in lesser insulin ground of a citizen. It is denoted by the notation M (S(.)) and
delivery and higher post prandial hyperglycemia. High post can be vague, imprecise, and can dynamically change.
prandial glucose levels is often managed through phantom Conceptualization of mental model of a citizen: Mental
carbs [Weaver and Hirsch, 2018], where the user announces model is in the user’s mind, but is embedded in the actions
a meal to the device, without actually consuming it. The pur- of the user for a given response from the controller. Hence, a
pose is to trick the device to administer a heavy bolus insulin. conceptualization of the mental model can be derived by ob-
An unpredictable unethical user behavior results in un- serving the action (external input X(t)) and response (output
resolved accountability: In user-centered design approach, Y (t)) in a deployed autonomous system.
the definition of ethical behavior is often unclear. This is be- The first step towards building a conceptualization of a
cause the user-centered design does not consider value sen- mental model is exposure analysis. It determines the subset
sitive aspects of an user, which are functions of the social E ⊂ X of exposures.
behavioral and economic background. As such the user may The second step is to derive a precise mathematical func-
often be unaware of ethically misaligned interaction. On the tion, E(t) = C(M (S(.)), S(X(t), t), Y (t)) that expresses
other hand, ASTS software has been only certified for the the temporal sequence of stressors applied by the user on the
specific use case. Since the interaction scenarios between the exposures. It characterizes the composite effect of the men-
user and the ASTS are potentially limitless, there may not be tal model, operational model, and the observed response from
specific guidance for a given ethically misaligned interaction. the controller on the inputs from the user.
For example, the case of phantom carbs is never mentioned Manifestations of mental model conceptualizations: In
in Minimed 670G user manuals or safety instructions. Since, this research, we will consider two types of manifestations:
the device is unaware of the status of the meal consumption, a) surrogate models, finite state machine, temporal logic or
such behavior can lead to severe hypoglycemia and poten- hybrid system based expressions of C(...) and b) task ac-
tially death. tion mapping models, that expresses the sequence of stressors
In an user centered design approach, when compliance from the user as a language.
fails due to an unexpected wrongful use case, accountability Example: Meal intake is manually managed in the Medtronic
becomes hard to resolve. 670G system. Typically a routine meal intake is expected and
the insulin delivery in the controller is dependent on this rou-
3 Socio-technical accountable software design tine. However, cultural/religious practices such as ramadan
can affect meal routines. Such changes in routines can be ex-
Social counter part consists of citizens as users, operators, pressed using a temporal logic model, where the finite states
oversight, certifiers and manufacturers. Whereas the tech- are events and the temporal properties express the change in
nical counter part consists of the software system. It is a meal timings for a given event.
co-design approach that involves iterative knowledge sharing The mental model that causes announcement of phantom
among two models: a) mental model that encodes the social meal can be conceptualized using a state machine based sur-
aspects of the design and b) operational model that encodes rogate model that is similar to the operational model of the
the technical aspect of the design. Medtronic 670G but has an extra phantom meal mode to ex-
Operational Model: An autonomous system operational press fake carb entry.
characteristics is captured by the function Y (t) = S(X(t), t), Regulatory statute: It is a textual description of a property
where Y (t) is the response of the autonomous system and that the autonomous system should abide by. It is often a gen-
X(t) Sis the external input to the system. A susbset of eral guidance and is intended to apply for a broad set of citi-
X(t) Y (t) is monitored in the real time using sensors de- zens with varied mental models. Compliance with regulation
ployed with the system. can be qualified by limiting the context of usage that includes
Manifestations of operational model: The software of an external environment and responses from the citizen.
autonomous system can be abstracted using various formal Regulatory Requirements: A regulatory statute can be in-
structures such as finite state machine or hybrid systems. terpreted as a envelop (max, min) on the observed output vari-
Such models can express the input output relationship in an ables (Y (t)) of an autonomous system for a specific use case
autonomous software by combining discrete modes and dy- taking into account the mental model of the citizen, and exter-
namic variations of system parameters over time. nal environmental context of usage. The autonomous system
Example: In case of the Medtronic 670G semi-autonomous is expected to meet the requirements in order to comply with
insulin control system, an operational model can be a hybrid the regulatory statute.
system. The finite state machine part of the hybrid system Accountability in software: Accountable software system
models the discrete modes such as basal, auto, correction bo- is a system that is flexible to change and context, express-
lus, and meal bolus modes of the software, whereas each state ibe enough to capture policy, provably and certifiably com-
Key performance indicator triggering event,
pliant, and transparent/auditable to policymakers and stake-
Visible
safety performance indicator triggering event,
holders while maintaining the manufacturer’s competitve ad- Predict
Law violation event, incident, or accident. L1
vantage by not revealing their trade secrets. In this proposal, Flow dynamics which represent holistic system’s
component interaction behavior.
we define accountable software systems as systems that en- L2
able stakeholders to ensure and prove compliance in the field Root-Cause Actual Model of the system which is a result of
of operation. Three properties constitute accountability: Analysis the actor’s Interaction with Actual System. This
Invisible
model is also influenced by the actor’s
a) operational transparency: The system software should interpretation theories (developed from culture
provision for: a) monitoring of input X(t) and response Y (t) and personal experiences).
L3
parameters, and b) an explanation interface that can extract Actor’s Mental Model of how the system works.
This is usually shaped by external information
relevant knowledge from the observed parameters for differ- including media, reports, manuals, policies, and
ent stakeholders in the software development, deployment, regulations. L4
certification, and regulation process.
b) operational adaptability: When a change in interpreta- Figure 3: Mental model levels.
tion of the regulatory statute results in new context sensitive
requirements, the software provides mechanisms to monitor
relevant variables necessary to evaluate compliance with the • Basic interpretation of Law: This should focus on
new requirements. what a given regulatory statute actually mean for a given
c) operational interpretability: In the event of compliance actor. This is irrespective of any ASTS solution in the
failure, the cause can be attributed to a specific software com- domain.
ponent or citizen interaction, or environmental conditions. • Interpretation for a given system: In this task, a thor-
ough description of the ASTS should be provided to the
4 Pillars of Socio-Technical Co-Design actor through video demonstrations and advertisements.
The actor should then be interviewed about their perfor-
Continuous monitoring is key to the socio-technical co- mance expectation from the system and their perception
design approach. The input output set (X(t), Y (t)) is mon- of compliance for the given system.
itored for a deployed software which is regulatory compli-
ant for the population average mental model. The following To interpret the perception of compliance from the inter-
pillars should then guide our socio technical co-design ap- view data a collaborative effort between the SBE and techni-
proach. cal experts can be undertaken utilizing the System Theoretic
Process Analysis (STPA) formalism [Leveson et al., 2003].
4.1 Pillar 1: Extraction of Tacit Knowledge STPA is a hazard analysis technique that provides guidance to
engineers in the design process and is widely used by indus-
Mental models are tacit knowledge for ASTS software and tries including autonomous vehicles, Advanced Driver As-
are layered as depicted by the Iceberg systems thinking model sistance Systems (ADAS), unmanned aerial vehicle, nuclear
(Figure 3) [Webb et al., 2008]. The lowest level (L4) is the power plants and many other safety-critical software systems
farthest from observable aspects of the software and repre- [Leveson et al., 2003]. STPA accounts for a broad range of
sents the actor’s assumptions about the software system be- causal factors including dysfunctional system interactions, in-
fore interacting with it. Such assumptions is built based on complete/incorrect actors’ mental models, and flawed design.
information acquired from reports, manuals, policies, regula- STPA has been extended to analyze causal factors of undesir-
tions, and media. This is also affected by the actor’s social, able events arising from flawed actors’ mental models. The
economic, and cultural background. actor’s mental model encompasses the mental model of the
L3 represents the refined mental model of the actor after environment (legal, social, and economical contexts) where
its interaction with the software. At this level, actors adapt the system operates, mental model of the system which is
(contradict or enhance) beliefs they have about the system built by the actor using information from reports, media, and
based on their own experience with the system. The result any educative documents, and the mental model of the sys-
of such an interaction can be interpreted differently from one tem’s expected behavior which describes the actor’s expec-
actor to the other depending on the theory they use to create tations of how the system will behave. The actor adapts the
meanings. expected behavior model based on sensory inputs and feed-
The next level L2 represents the trends and patterns of the back from real world interaction. STPA also analyzes how
holistic system’s operational characteristics arising due to the humans may adapt their mental models from interaction with
interaction of the actor. Information on this layer is typically the system and include the repercussions of flawed model
available to the manufacturer and is used to evaluate perfor- adaptations. As shown in Figure 4, the actions and behavior
mance and regulatory compliance. of the human-in-the-loop is influenced by the human’s inter-
Finally, L1 represents observed events that are caused by pretation of a variety of external inputs and how the human
the holistic system’s operational characteristics. assimilates the input information into mental model represen-
tations.
Value Sensitive Interpretation of Regulatory Law
This is tacit knowledge extraction from Level L4. There are Conceptualization of Mental Models
no observable parameters that can be utilized to derive or val- Information from Level L3 can be utilized to conceptualize
idate the mental model of the actor. However, there are a rich mental models. The L3 information is in the form of exter-
body work in the social sciences domain that have already nal inputs to the ASTS obtained from the actors in a context
studied various social systems. rich environment. Understanding the learning model of the
Potential Methodology: Social values guide an actor’s per- actor is a significant step towards conceptualization of men-
formance expectation as well as perception of compliance to tal model. Literature in human computer interface (HCI) re-
regulatory law. User engagement should involve interviews search suggest that there are eight different types of human
to identify two aspects: learning models [Gentner and Stevens, 2014] -
physiological factors
(e.g., stress, fatigue)
On the other hand, if the meal management strategy leads
Human-in-the-loop to prolonged hyperglycemia, then the Minimed 670G trig-
Human’s Mental Model (MM)
gers the correction bolus mode. In this mode, the actor is
Adapt
MM of the System’s
Interpret
Sensors’ information, Displays, prompted to take a Glucosemeter reading and enter it to the
Actions
Adapt
Expected Behavior and Feedback
system. Here the response set Y (t) causes the input set X(t).
MM of the System
Interpret Reports, Media,
Manuals, Procedures
An essential step towards decoupling the mental model
Adapt
MM of the Interpret Physical, Social, with the operational model is to derive the causal relation-
Environment Legal, Cultural, and Economical
Contexts
ships between X(t) and Y (t). These causal relationships can
be obtained from an algorithmic description of the ASTS.
1. Identify
Unsafe 2. Identify MM Flaws 3. Identify MM Adaptation Flaws
4. Identified
Causal
4.2 Pillar 2: Certification game
Actions Scenarios
The purpose of this pillar is to evaluate compliance for new
Figure 4: STPA Extension to Include Human Factors.
modes of operation or under the new value sensitive require-
ments. However, the inherent assumption is that all required
data will be monitored and shared between stakeholders. This
a) Strong Analogy, where the actor finds a strong similarity to is often not feasible, because of the sensitive nature of data
another software that they have prior experience with. and potential violation of regulatory laws such as HIPAA,
b) Surrogate models, where the actor derives notational ana- GDPR, and patent laws. The operational adaptability compo-
logue such as a finite state machine model of the mechanism nent of accountable software necessitates that the manufac-
of the software. turer participate in a certification game with the certifier. The
c) Mapping models, where the actor makes a table of actions objective of the game for both manufacturers and certifiers is
and responses. to evaluate the compliance of either: a) the new mode of op-
d) Coherence models, where the actor makes a logical schema eration, or b) the present mode of operation under novel value
of operational characteristics of the software which helps the sensitive requirements. The game consists of a sequential ex-
actor to remember how to interact. This model is vague be- ecution of steps initiated by manufacturer and continued by
cause if the response of the software is not coherent with the the certifier.
schema, the software feature maybe forgotten. Step 1: Manufacturer Share Level L1 information. This
e) Vocabulary model, where the actor creates a grammar that information sharing is a part of continuous compliance check
expresses the temporal sequence of actions that are required property of accountable software.
to be performed by the actor to elicit a given response. Step 2: Certifier checks learnability: The certifier then at-
tempts to mine tacit knowledge using theories in Thrust 1.
f) psychological grammar, where the actor finds an analogy The result of this step is a guidance to the manufacturer re-
with the grammar of their native language. garding potential sharing of Level 2 and Level 3 information.
g) Problem space, where the responses of the actor is modeled Step 3: Certifier evaluates compliance properties: With
as a solution to challenge question from the software. the information currently available to the certifier, it uses
h) Commonality model, where actor actions are considered as formal software analysis tools and techniques to extrapolate
processes sharing the same data structure. compliance properties to the new mode of operation, or to the
Potential Methodology: Data collected from each interview present mode of operation with value sensitive requirements.
conducted should be utilized to instantiate the conceptualiza- Step 4: Manufacturer Share Level L3 or L2 information.
tions with appropriate models. The models have to be un- The manufacturer takes the learnability and compliance guid-
ambiguous and complete. State reachability analysis can be ance and makes a decision to lawfully share L3 or L2 infor-
utilized to check for undefined reachable states. mation to the manufacturer. In the process the manufacturer
Decoupling mental model and operational model performs a cost risk and benefit analysis and may chose to
The operational characteristics of an ASTS is a composite share a different set of information than that required by the
result of a closed loop execution of two processes: a) mental certifier or nothing at all.
model guided interaction of the actor with the software, and Step 5: Repetition of Step 2 by Certifier. If new informa-
b) response of the software to the actor’s input. The operation tion is shared the certifier repeats Step 2. If no information
is almost always initiated by the actor, which is sensed using is shared, the compliance extrapolation obtained in Step 3 is
sensors, the software then reacts to the input by generating issued as guidance to the actors.
a response, that is provided as feedback to the actor, closing Evaluating Learnability of Tacit Knowledge
the loop. As such this two way causal relationship can be An ASTS design consists of an actor model that embeds
extracted by analyzing the dependencies of the input set X(t) the assumed interactive behavior of the actor with the re-
and response set Y (t). maining components of the system, an environment model
Potential Methodology: The problem is an evolved form usually represented by a set of ordinary differential equa-
of system identification where the observable variables are tions (ODEs) governing the high-dimensional system, and the
controlled by two parallel co-operating processes. For the controller model that utilizes participants and environment
Medtronic 670G example, the meal management software is models along with sensor data to determine control actions
initiated by the actor by providing a carbohydrate value as to satisfy a predefined goal function. We consider the fact
an input. The bolus wizard software component then com- that contingencies in participant behavior and novel/unseen
putes the amount of insulin to infuse and administers a bo- Environment-Controller-Actor interaction scenarios can be
lus insulin. Subsequently, based on the continuous glucose detected as a deviation from the expected evolution of the sys-
monitor (CGM) sensor values it executes a PID control strat- tem’s dynamics. An ultimate solution to enhance safety is to
egy to continually infuse micro bolus insulin to control the learn changes in the variation of the physical dynamics within
blood glucose. For this operational context the input set X(t) each controller mode and verify whether theses changes rep-
causes the output response Y (t). resent a potential hazard to the system using state-of-the-art
safety verification techniques that are employed during the mode. Research in socio-technical co-design of ASTS soft-
system’s engineering[Henzinger et al., 1997]. ware will result in an improved performance of autonomous
Potential Methodology: An intuitive approach to solve this systems in practical deployments and providing higher com-
problem is to directly use residual neural networks or ODE pliance guarrantees to stake holders. This approach relies on
nets to learn a generative latent model of the dynamical sys- collaboration between CSE and SBE scientists with the aim
tem. Although deep learning techniques can learn model pa- of analyzing the bi-directional impact between the software
rameters but they require large amount of data which may not system’s operation and the legal, social, and behavioral con-
be available in practical deployment scenarios. The complex- texts of the system’s operating environment.
ity and resources required for learning such models is propor-
tional to the number of function evaluations performed in the References
forward pass, i.e. the size of the governing ODEs, number of [Ahmed et al., 2020] S. H. Ahmed, D. L. Ewins, J. Bridges,
unknown parameters, and number of latent variables. Such A. Timmis, N. Payne, C. Mooney, and C MacGregor. Do-it-
large I/O data may not be available due to several reasons yourself (diy) artificial pancreas systems for type 1 diabetes: Per-
including data logging insufficient capacity or a high learn- spectives of two adult users, parent of a user and healthcare pro-
ing frequency required for the safety evaluation of the semi fessionals. Advances in therapy, 37(9):3929–3941, 2020.
autonomous system. Another approach is to utilize contex- [Berget et al., 2020] Cari Berget, Laurel H Messer, Tim Vigers,
tual conditions to reduce the model learning to a set of linear Brigitte I Frohnert, Laura Pyle, R Paul Wadwa, Kimberly A
or polynomial regression analysis. Contextual information of Driscoll, and Gregory P Forlenza. Six months of hybrid closed
data can provide initial and asymptotic conditions of every loop in the real-world: An evaluation of children and young
operational mode to simplify the operational model learning. adults using the 670g system. Pediatric diabetes, 21(2):310–318,
2020.
4.3 Pillar 3: Reshaping mental model [Ferenci et al., 2015] Tamás Ferenci, Anna Körner, and Levente
Kovács. The interrelationship of hba1c and real-time continuous
The third arm of accountability in software is interpretabil- glucose monitoring in children with type 1 diabetes. Diabetes
ity of compliance properties to the stake holders. Given the research and clinical practice, 108(1):38–44, 2015.
diverse goals and backgrounds of the stake holders, effective [Gentner and Stevens, 2014] Dedre Gentner and Albert L Stevens.
feedback should be a function of the objectives of each stake Mental models. Psychology Press, 2014.
holder and should not be specific to an instance of the oper- [Graafstra et al., 2010] A. Graafstra, K. Michael, and M. G.
ation of the ASTS software. Concept level feedback to stake Michael. Social-technical issues facing the humancentric rfid im-
holder is essential for better communication. In the process of plantee sub-culture through the eyes of amal graafstra. In 2010
forming a mental model of any system or process, the human IEEE International Symposium on Technology and Society, pages
learns general concepts that is applicable to any instance of 498–516, 2010.
the system or process. According to human learning theories, [Henzinger et al., 1997] Thomas A Henzinger, Pei-Hsin Ho, and
a feedback in terms of concepts used is effective in creating Howard Wong-Toi. Hytech: A model checker for hybrid sys-
memories and taking actions to complete objectives. tems. International Journal on Software Tools for Technology
Transfer, 1(1-2):110–122, 1997.
Feedback to Manufacturer [Leveson et al., 2003] Nancy G Leveson, Mirna Daouk, Nicolas
The certifier in the certification game provides feedback to the Dulac, and Karen Marais. Applying stamp in accident analysis.
manufacturer in terms of the operational model and its com- 2003.
pliance properties. However, such feedback may not directly [Michael et al., 2021] K. Michael, R. Abbas, R. A. Calvo, G. Rous-
enable the manufacturer to identify the components that can sos, E. Scornavacca, and S. F. Wamba. Smart infrastructure and
be monitored or adapted to facilitate compliance evaluation technology systems ethics. IEEE Transactions on Technology
and satisfaction. The manufacturer is familiar with the soft- and Society, 2(1):2–3, 2021.
ware design and typically develops architectural models of [Robertson et al., 2017] L. Robertson, A. M. Aneiros, and
the software before implementation. Hence if a feedback K. Michael. A theory of exposure: Measuring technology system
from the certifier is in terms of the these architectural model end user vulnerabilities. In 2017 IEEE International Symposium
components then it is one step closer to identifying the next on Technology and Society (ISTAS), pages 1–10, 2017.
steps towards accountability. [Robertson et al., 2019] Lindsay J Robertson, Roba Abbas, Gursel
Alici, Albert Munoz, and Katina Michael. Engineering-based
Value Sensitive feedback to actor design methodology for embedding ethics in autonomous robots.
Contrary to the manufacturer, feedback to the actor may not Proceedings of the IEEE, 107(3):582–599, 2019.
be concretely expressed in terms of some objective compo- [Truong et al., 2019] Nguyen Binh Truong, Kai Sun, Gyu Myoung
nents such as software code. While feedback to the manu- Lee, and Yike Guo. Gdpr-compliant personal data management:
facturer is uniform, for an actor the feedback should be di- A blockchain-based solution. IEEE Transactions on Information
verse commensurate with their social, behavioral, economic Forensics and Security, 15:1746–1761, 2019.
and cultural background. [Weaver and Hirsch, 2018] Kathryn W Weaver and Irl B Hirsch.
The hybrid closed-loop system: evolution and practical applica-
tions. Diabetes technology & therapeutics, 20(S2):S2–16, 2018.
5 Conclusions [Webb et al., 2008] David C Webb, Nina Boswinkel, and Truus
Autonomous systems are failing to maintain and ensure com- Dekker. Beneath the tip of the iceberg: Using representations
pliance to regulations when deployed in practice resulting to support student understanding. Mathematics teaching in the
in loss of public trust. The Boeing 737 Max 8 has been middle school, 14(2):110–113, 2008.
grounded worldwide, increasing incidence of crashes involv- [Winfield et al., 2019] A. F. Winfield, K. Michael, J. Pitt, and V. Ev-
ing autonomous cars resulting in lawsuits against companies, ers. Machine ethics: The design and governance of ethical ai and
and average Type 1 diabetic subjects having a decreasing autonomous systems [scanning the issue]. Proceedings of the
amount of time that they are spending in closed loop auto IEEE, 107(3):509–517, 2019.