Addressing Early Life Cycle Privacy Risk Applying System-Theoretic Early Concept Analysis and Model-Based Systems Engineering to Privacy Stuart S. Shapiro The MITRE Corporation Bedford, MA USA sshapiro@mitre.org Abstract—This paper adapts System-Theoretic Early Concept case with STPA.) The “analysis” in STECA is deceptive, Analysis (STECA), an instrumental safety risk management though, as the target of that analysis is intended to be a concept technique, for privacy to better identify and address privacy of operations (ConOps), a natural language description of the risks early in the engineering process. The technique, STECA- system’s operation, which provides inputs to a more Priv, aims to infer a nominal functional privacy control structure fundamentally instrumental process of postulating an based on a conceptual system description and privacy-related appropriate control structure. system behavioral constraints. Model-based systems engineering (MBSE) is employed in conjunction with STECA-Priv to In addition to adapting STECA for privacy (yielding validate the projected control structure and to identify privacy STECA-Priv) in much the same way we adapted STPA for risks in the form of constraint violations. To illustrate STECA- privacy (yielding STPA-Priv), we leverage model-based Priv as supported by MBSE, it is applied to the simplified systems engineering (MBSE) as a mechanism for checking the example of a smart television. integrity of the postulated control structure. (STECA itself was partially inspired by MBSE [4].) In MBSE, complex systems Keywords—privacy risk; System-Theoretic Early Concept are represented as models using diagrammatic modeling Analysis; STECA; STECA-Priv; model-based systems engineering; languages such as Unified Modeling Language (UML) or MBSE Systems Modeling Language (SysML) and current tools render these models executable. This presents opportunities for I. INTRODUCTION detection of unanticipated emergent properties and helps Engineering invariably involves both analytical and ensure consistent and up-to-date life cycle documentation, instrumental methods. The former tend to be more since the model is used to generate these documents. The straightforward than the latter as they target an extant situation International Council on Systems Engineering (INCOSE) and or specification, analyzing something that in some shape or the Object Management Group (OMG) [5] are among the form already exists. Instrumental methods, in contrast, support organizations driving the development of MBSE and it has the creation of something new. It is not surprising, then, that been adopted for systems engineering by organizations such as methods for managing privacy risk as part of socio-technical the Jet Propulsion Laboratory (JPL) [6]. MBSE in combination system design are thinner on the ground than methods for with STECA-Priv constitutes a potentially powerful tool-based assessing risk in already designed (and possibly already approach to the risk-aware design of privacy-sensitive systems. implemented) systems. This is not to imply the adequacy of The remainder of the paper is organized as follows. Section current methods for privacy risk analysis, but rather to note that II provides background on STAMP, STECA, and MBSE. however problematic the state of analytical privacy risk Section III discusses STECA and the modifications for privacy management techniques, the state of instrumental privacy risk that result in STECA-Priv. Section IV uses the example of a management techniques is even more so. smart TV to illustrate the combined use of STECA-Priv and Previously [1], we sought to add to the stable of analytical MBSE. Section V situates STECA-Priv with respect to some privacy risk management techniques by adapting a related work while Section VI presents some concluding methodology [2] initially developed to assess safety risk in thoughts. support of safety engineering—System-Theoretic Process Analysis (STPA), grounded in System-Theoretic Accident II. BACKGROUND Model and Processes (STAMP)—and later extended to assess security risk in support of security engineering [3]. Here, we A. STAMP aim to do the same for instrumental privacy risk management STAMP frames safety in terms of constraints rather than techniques by adapting another STAMP-related technique, events [2]. Safety is achieved through the proper enforcement System-Theoretic Early Concept Analysis (STECA), of complete and correct constraints on system behavior, rather developed for use in the early stages the systems engineering than the prevention of certain events or chains of events. The life cycle (SELC) [4]. (Extensions of STECA supporting other more inclusive notion of behavioral constraints has the types of engineering doubtless are possible, as has been the potential to identify problems arising out of issues such as Approved for Public Release; Distribution Unlimited. Case Number 17- 0621 unanticipated component interactions. These constraints are systems engineering from a document-centric to a model- what controls enforce. centric approach, a reaction to the effort required to develop and maintain traditional SELC documentation, including Controls are structured hierarchically with controls at each requirements and design specifications, as well as to the level enforcing constraints on processes in the level below it. increasing difficulty of understanding complex socio-technical This control structure exhibits multiple aspects. As described systems at even a conceptual level. In MBSE, the model serves by Leveson [2], controls invariably involve adaptive feedback as the central artifact and traditional SELC documentation is mechanisms, i.e., they are closed-loop controls. (Some privacy automatically generated from the model, more efficiently and controls, though, lack feedback loops, i.e., they are open-loop effectively maintaining currency and consistency. Simulation controls.) Communication channels carry control commands to via executable models enables MBSE to be leveraged across the relevant processes and information from the processes to the SELC. the controllers. Accidents can result from four different types of control errors: A variety of powerful tools are available that support MBSE using various modeling languages, including UML and • Incorrect control action SysML. For our purposes, SysML is preferable due to how it • Missing control action handles constraints. Constraints in UML (specified using Object Constraint Language) are annotations, i.e., they convey • Control action provided at the wrong time information to the engineer but are not operationally integrated within the model. SysML constraints, in contrast, are integrated • Incorrect duration of control action into model execution so that as it executes, constraint For a control to properly constrain a process, it must violations can be observed and captured. maintain a model of that process. Control errors arise when the This capability is essential to extracting full benefit from process model being used by a controller doesn’t properly using MBSE in conjunction with STECA. MBSE using SysML correspond to the process being controlled. (This can happen will reveal aspects of the projected control structure that could either because there is an error or gap in the model or because potentially result in constraint violations and thus present risks. the model’s state does not match the actual process state.) Those aspects of the projected control structure can then be reconsidered and the risks represented by the constraint B. STPA violations appropriately managed through mitigation, STPA is a safety risk analysis methodology based on the avoidance, transfer, or explicit acceptance. Any resulting concepts of STAMP. It was adapted for security (STPA-Sec) changes to the control structure are captured by the model and [3] prior to being adapted for privacy [1]. Its application in all their effects verified through its execution. its variants, however, requires a reasonably fleshed out system specification or description, such that the relevant control III. FROM STECA TO STECA-PRIV structure can be extracted and analyzed. This presents obvious problems when in the early, conceptual stages of the SELC. As previously noted, STECA aims to project a system STECA [4] is a response to this problem and aims to infer a control structure from a description of system behavior and a required control structure by extracting control concepts from safety risk model expressed in the form of system behavioral an early-stage system description, specifically a ConOps that constraints. Developed by Fleming [4], STECA consists of six describes envisioned system behavior at a high level without major (not strictly linear and potentially iterative) steps divided specifying how that behavior will be achieved. STECA into ConOps analysis and safety-driven design: effectively attempts to deconstruct the system description to 1. Identify system hazards (ConOps analysis) determine how applicable behavioral constraints could be enforced. Whereas STPA is fundamentally an analytical 2. Derive system safety constraints (safety-driven design) technique (answering the question “What is going on here?”), 3. Identify control concepts (ConOps analysis) STECA is fundamentally an instrumental one (answering the question “What should be done here?”). 4. Identify hazardous scenarios and causal factors (ConOps analysis) C. MBSE 5. Derive refined safety constraints (safety-driven design) Because STECA aims to project a nominal functional control structure consistent with the system description and 6. Refine, modify control structure (safety-driven design) relevant constraints, it lends itself to MBSE as a means of In the same way that STECA leverages the activities describing and assessing that control structure. MBSE has been developed for STPA, in adapting STECA for privacy we can driven in part by a 2007 joint initiative of INCOSE and OMG avail ourselves of the adaptations of STPA developed for [6]. This initiative (part of the larger INCOSE SE Vision 2020) STPA-Priv. Indeed, the first several steps are largely identical defines MBSE as “the formalized application of modeling to to STPA-Priv (and we refer the reader to [1] for detailed support system requirements, design, analysis, verification and explanations of them): validation activities beginning in the conceptual design phase and continuing throughout development and later life cycle 1. Identify potential adverse privacy consequences to be phases” and notes the development of mathematical considered, as denoted by a selected framework foundations in 1993 [7]. MBSE constitutes an effort to shift 2. Identify vulnerabilities that can lead to adverse privacy TABLE I. STECA-PRIV VERSUS STECA consequences in the context of the system STECA-Priv STECA 3. Specify system privacy constraints Identify potential adverse privacy Identify system hazards consequences to be considered, as As with STPA-Priv, we structure STECA-Priv to denoted by a selected framework accommodate any of a variety of privacy frameworks (i.e., risk Identify vulnerabilities that can lead models), recognizing the pluralism of understandings of to adverse privacy consequences in privacy in this regard. the context of the system Specify system privacy constraints Derive system safety constraints In STPA-Priv, Step 3 combines the specification of privacy Identify system privacy control Identify control concepts constraints with the specification of the system functional concepts and infer privacy control control structure. However, for STECA-Priv we must infer that models structure, a less straightforward process. It is at this point, Use MBSE to represent privacy Identify hazardous scenarios and therefore, that STECA-Priv substantively departs from STPA- control structure and constraints as causal factors an executable model to identify risks Priv: in the form of constraint violations 4. Identify system privacy control concepts and infer and their causal factors, including privacy control models malicious actions Revise privacy control structure and Derive refined safety constraints 5. Use MBSE to represent privacy control structure and constraints Refine, modify control structure constraints as an executable model to identify risks in the form of constraint violations and their causal factors, including malicious actions The smart TV will collect and store a record of the programs watched on the device. This viewing data will consist OR of entries that include the date and time (timestamp), the channel number selected, the program name, and the service 5. Manually analyze privacy control structure for provider (to enable matching the channel number to a specific a. Completeness network). This viewing data will be regularly transmitted to the TV manufacturer, who will combine it with demographic data b. Allocation of system privacy responsibilities (based on IP address) and maintain the combined data, c. Coordination and consistency including IP address, in a repository. Users must opt-in (provide explicit consent) to the collection and use of viewing 6. Revise privacy control structure and constraints data by the manufacturer, as governed by the privacy policy Note the absence of explicit reference to a ConOps. While associated with the TV. one could be used for STECA-Priv if it exists, it is not strictly For the purposes of this example, the preceding paragraph necessary. (Nor, arguably, is it strictly necessary for STECA). will function as the system description that serves as input to While substituting other types of high-level system description STECA-Priv. (nominal use cases or data flows, for example) may introduce additional difficulty, it does not fundamentally change the A. Step 1: Identify potential adverse privacy consequences to approach. By the same token, opting to represent the control be considered, as denoted by a selected framework structure and privacy constraints as an executable model doesn’t fundamentally change the approach either. Rather, it As we did when illustrating the use of STPA-Priv enhances it by enabling the identification and management of previously, we will use Calo’s subjective/objective privacy the privacy risks represented by constraint violations, a more harms [8] as the framework for identifying adverse privacy focused lens for viewing the conceptual system than the less consequences. Our choice of this framework is motivated by its precise characteristics in the alternative Step 5. Table I shows relative simplicity, a useful characteristic for demonstration how the steps of STECA-Priv compare with the steps of purposes. For a real-world project, a different or additional STECA, assuming use of MBSE. framework, including Fair Information Practice Principles (FIPPs), almost certainly would be used. A subjective privacy harm is the perception of unwanted surveillance. An objective IV. APPLYING STECA-PRIV privacy harm is the forced or unanticipated use of personal As before [1], we will use the example of a smart television (i.e., specifically related to a person) information. to illustrate the application of the methodology. However, unlike the STPA-Priv example, which analyzed an existing B. Step 2: Identify vulnerabilities that can lead to adverse smart TV implementation (as inferred from its privacy policy), privacy consequences in the context of the system STECA-Priv will be used to support the conceptual stage of The use of explicit consent goes a long way toward engineering a smart TV. To keep the example small, we will avoiding the potential for subjective privacy harms. However, focus on a distinct subset of the TV’s operations: the collection and use of viewing data, managing consent for this collection an objective privacy harm may result if that consent isn’t accompanied by an accurate governing privacy policy that and use, and managing the governing privacy policy on the conveys the collection and use of viewing data and any device. relevant terms (e.g., regarding user access to the collected the control loop can achieve the four necessary conditions2 of process control and ade- quately interact with its environment, other processes, and other controllers. In other words, these guide words are necessary to ensure that a control loop is controllable and coordinable with other controlled processes. data). Similarly, an objective privacy harm may result if 9. Control input 8. Feedback to higher consent is not obtained or renewed following a change to that (setpoint) or other level controller commands policy. This may be accompanied by a subjective privacy harm 11. External 10. Controller input output upon realization by a user of a material change to the policy 1. Controller 7. Control 6. Control 5. Process absent consent. The relevant vulnerabilities can be expressed Action Algorithm Model as: 2. 4. 1. The privacy policy associated with the smart TV is Actuator Sensor inaccurate as it pertains to viewing data collection and use. 12. Alternate 3. Controlled Controller 2 control actions Process 2. Privacy policy and viewing data consent become 13. External 15. Process unsynchronized. process input 14. Process output disturbance C. Step 3: Specify system privacy constraints These vulnerabilities can be reframed as corresponding Figure Fig. 1. 11:STAMP Control Control Loop with generic Loop [4] entities system privacy constraints: OnceThe information sets of inthese Figure 11 and the above elements lists (Controller, have Actuator, Controlled been identified, the 1. At any point in time, the privacy policy associated Process,control privacy Sensor) canmodel then be used to systematically to corresponding parseeach and query setthecan natural belan- with the TV must be accurate. guage description or graphical depiction in a concept of operations. The resulting developed. (It is not expected that all potential model elements This can be decomposed into two distinct constraints: willmodel and subsequent database are easy to interrogate and visualize. These quali- be specified; not all elements are mandatory in a control ties help the analyst to check for internal inconsistencies and/or missing information loop and limited information may preclude identifying or that may result in unsatisfied control conditions, and also to check for inconsistencies a. The privacy policy resident on the smart TV is projecting others.) Table II provides questions to guide this across the system hierarchy. the current privacy policy in effect for the smart process. This yields the control models in Tables III and IV, Table 6 provides a series of prompts that an analyst can use when reading a text or TV. addressing constraints 1.a and 1.b respectively, and Tables V graphic in a ConOps. and VI, addressing constraints 2.a and 2.b respectively. To save b. The current privacy policy in effect for the smart In order to obtain a “complete” model of the ConOps, this model development space, these tables only contain those rows with populated TV correctly describes the applicable privacy approach should be applied recursively over the entire ConOps document. The key- descriptions. practices. words, with associated questions and comments (Tables 6 and 7), can be applied to Having 2 See page 56.inferred these control models, we can now use 2. Consent must correspond to the current privacy MBSE to represent the functional privacy control structure of policy. these aspects of the system together 61 with the applicable privacy This also can be decomposed into two distinct constraints: constraints. To keep the example manageable, we will focus on the control loops implied by Tables V and VI in which the a. Consent must be specific to the current smart TV smart TV is the controller. These control models address privacy policy. constraint 2. b. Viewing data may be transmitted only if consent has been registered. E. Step 5: Use MBSE to represent control structure and To this point, the process has been the same as if we were constraints as an executable model to identify risks in the applying STPA-Priv to the described system. But, of course, form of constraint violations and their causal factors, this is not a fully described or specified system, but a including malicious actions (circumscribed) high-level concept which must be fleshed out. We begin this process by creating the overall structure of It is at this point that STECA-Priv diverges from STPA-Priv. the model. Figure 2 shows this structure using SysML block definition diagrams. The principal blocks represent the smart D. Step 4: Identify system privacy control concepts and infer TV and the smart TV manufacturer. (Although denoted privacy control models separately, the TV viewing dataset is part of the smart TV manufacturer.) We have also included the smart TV user and In the absence of a more or less complete system specification, the demographics provider for context. (As a practical matter, we must infer a functional privacy control structure rather than the modeler will play the role of the user.) The specified extract the control structure. The initial step in performing this multiplicities are nominal and do not affect the example. inference is identifying the system privacy control concepts Communications between the user (actually the modeler) and from the limited description that we have. To accomplish this, the TV and the TV and manufacturer are implemented by we assign control loop roles to applicable elements in the SysML signals and events via ports that are denoted on internal system description. Figure 1 shows the general form of a block diagrams representing the TV and manufacturer blocks. STAMP control loop. A STAMP control loop includes four principal roles: controller, actuator, controlled process, and The remaining block is a constraint block, Consent-Policy sensor. The controller issues control actions, which are Sync. Constraint blocks define conditions that should hold at executed by some kind of (not necessarily physical) actuator, all times. As long as the constraint expression evaluates to true, which acts on the controlled process, the relevant behavior of the constraint is satisfied. If the expression evaluates to false, which is conveyed back to the controller by the sensor. TABLE II. STAMP CONTROL MODEL ELICITATION [4] TABLE V. PRIVACY CONSTRAINT 2.A CONTROL MODEL Control Role Description Control Role Description 1. Controller Which controller is being described in the 1. Controller Smart TV text? 3. Controlled Process Privacy policy update 5. Process Model Policy updated indicator (policyUpdated) 6. Control Algorithm Determine consent if privacy policy is 2. Actuator What mechanism(s) does the control have updated by manufacturer in order to affect the process? 7. Control Actions Receive policy update; process policy update 3. Controlled Process What process does the controller have control over? 9. Control Input Smart TV privacy policy 4. Sensor What type of feedback does the controller 12. Alternate Control Grant/deny consent receive about the process it controls? Actions 5. Process Model What states and variables does the 15. Process Output Displayed privacy policy and consent option controller know about the process it controls? TABLE VI. PRIVACY CONSTRAINT 2.B CONTROL MODEL 6. Control Algorithm Does the controller use an algorithm or procedure to generate action? Control Role Description 7. Control Actions What types of action can the controller generate? 1. Controller Smart TV 8. Controller Status Does the controller provide feedback to 3. Controlled Process Viewing data transmission higher level controllers? 5. Process Model Consent indicator (consent) 9. Control Input Does the controller receive set points or 6. Control Algorithm Periodically send viewing data if consent has other types of commands? been granted 10. Controller Output Does the controller have output other than 7. Control Actions Send viewing data through the actuator? This often includes 11. External Controller Consent granted/denied transmission of information to other Input controllers. 15. Process Output Viewing data | consent granted; no output | 11. External Controller Does the controller receive external input, consent denied Input either in terms of other system information or other controller action(s), or other (e.g., a power source)? the constraint has been violated. While primarily intended to 12. Alternate Control Does the process receive action from apply to models involving substantial mathematical Actions controllers other than in items 1 and 2? 13. External Process Input Does the process require external input to calculation, we are using this facility to define a logical function? Examples include pressure, constraint using Boolean variables, which are described below. power, and heat. The control models described by Tables V and VI are 14. Process Disturbance What environmental factors does the represented by the state machine associated with the smart TV process interact with? 15. Process Output Does the system require that the process block and shown in Figure 3. As a practical matter, control output something to other components? models and their representations may require interative (e.g., power, pressure) refinement. Within the control representation, control actions take the form of SysML activities. Thus, the activities called in TABLE III. PRIVACY CONSTRAINT 1.A CONTROL MODEL the state machine diagram correspoind to the similarly named control actions described in Tables V and VI. Control Role Description 1. Controller Smart TV manufacturer The process models corresponding to constraint 2 are 2. Actuator Server represented using Boolean variables whose values indicate 3. Controlled Process Update smart TV privacy policy whether a privacy policy update has been received from the 6. Control Algorithm Transmit smart TV privacy policy when smart TV manufacturer (2.a) and whether consent for the current policy changes collection and use of viewing data has been granted (2.b). To 7. Control Actions Transmit current smart TV privacy policy avoid viewing interruption, policy updates are presented to the 15. Process Output Smart TV privacy policy user when the TV is turned on, requiring two distinct control actions for receiving and processing policy updates. Note that TABLE IV. PRIVACY CONSTRAINT 1.B CONTROL MODEL because a policy update is registered as the default value Control Role Description (policyUpdated == true), this will happen “out of the box.” 1. Controller Smart TV manufacturer If policyUpdated == true, the TV channel is switched to 2. Actuator Data governance compliance process channel 0 (the TV’s user messaging channel) to display the 3. Controlled Process Smart TV privacy policy implementation updated policy and the policy update indicator is cleared 6. Control Algorithm Periodic and triggered review (policyUpdated = false). The user must then grant or deny 7. Control Actions Revise smart TV privacy policy to conform to consent, which sets the consent indicator accordingly. The TV practices then enters its Operating state. (If policyUpdated == false, the 15. Process Output Smart TV privacy policy TV moves immediately from the On to the Operating state.) If consent has been granted (consent == true), the TV Fig. 2. Structure of Smart TV Model Fig. 3. Smart TV State Machine Diagram periodically transmits viewing data to the manufacturer, update indicator is set (policyUpdated = true). This where it is processed and stored. Thus, policyUpdated acts as prevents viewing data from being transmitted under the new a transition guard in the On state while consent acts as a policy until the user has had the opportunity to grant or deny transition guard in the Operating state. While in the consent. This is verified by making the necessary changes Operating state, the TV monitors for privacy policy updates executing the modified model, and observing that the and, if one is received, replaces the previous policy with the constraint failure no longer occurs (and the relevant variables new one and sets the policy update indicator (policyUpdated in the right window of Figure 4 are shaded green rather than = true). red). The constraint reflects the fact that at no time should there be an updated policy indicator (policyUpdated == V. RELATED WORK true) together with affirmative consent (consent == true). If STECA-Priv, like STPA-Priv, bears some relationship to this occurs, it implies that an updated privacy policy has not goal-oriented modeling [9], since it implicitly deals with yet been presented to the user for them to consent to, but that goals in the form of constraints and obstacles in the form of consent has nevertheless been granted. This would constitute problematic control actions. Further, there has been a violation of constraint 2.a. Note that constraint 2.b is discussion of leveraging MBSE in support of goal-oriented enforced by the guard condition on the Operating state modeling [10], though it is unclear what, if anything, has transition that results in the transmission of viewing data. resulted from this. There has also been work using modeling As the left window of Figure 4 shows, this constraint was to support Privacy by Design based on a specific privacy violated (“failed”) when the model was executed. Execution framework [11], including automating OCL constraint involved turning the TV on, granting consent, then updating checking [12]. the privacy policy. The right window shows the implicated Also like STPA-Priv, but unlike goal-oriented modeling, variables shaded red. (While the constraint held, they were STECA-Priv takes an approach explicitly grounded in shaded green.) Upon examination, the problem becomes systems theory. This manifests itself not only in a typology apparent. Because an updated privacy policy is not of problematic control actions, but also in the use of role- immediately presented to the user and the consent indicator based control models. Together with the use of a chosen is not cleared when the policy update indicator is set, privacy framework, these contribute to a more focused and viewing data continues to be transmitted. In other words, if practically-structured methodology. As such, it is a the user had previously consented to the collection and use methodology that strikes a balance between rigidity and of viewing data by the manufacturer, that consent would open-endedness. continue under a new policy until such time as the TV was turned off and then on again. This balance is also reflected in the use of MBSE in support of STECA-Priv. By definition, MBSE is a general, open-ended process, as is reflected by some of its specific F. Step 6: Revise privacy control structure and constraints methods [13]. However, because the control models derived From a STAMP perspective, this represents a control from the application of STECA-Priv drive the MBSE action provided at the wrong time. The most straightforward process, it is circumscribed in a way not characteristic of way of addressing this (thereby mitigating the privacy risk MBSE methods more generally. However, owing in part to related to the constraint violation), is to clear the consent the limits of reasonable inference, there still will be indicator (consent = false) at the same time as the policy Fig. 4. Constraint Violation During Model Execution significant degrees of freedom available to the engineer. representation involves a translation from the relative Striking this balance, one way or another, is historically abstractions of the STECA-Priv control models to the more necessary for effective engineering praxis [14] within a given specific constructs of SysML. Such guidance actually might engineering discipline. Insufficient focus leaves practitioners be considered a subset of broader guidance regarding how to struggling for traction, while an overly prescriptive approach perform that translation. If such guidance can be developed, suffers from limited efficacy. the application of STECA-Priv in conjunction with MBSE would become less of an art and more of a discipline. VI. CONCLUSION STECA-Priv is an instrumental method for performing ACKNOWLEDGMENT privacy risk management on complex socio-technical Thanks to MITRE colleague Julie Snyder and to an systems at early life cycle stages. It does not require a anonymous reviewer for comments that signficantly relatively complete system description, instead inferring improved this paper. provisional privacy control structure in a systematic fashion for the purpose of system specification. Note, though, that REFERENCES the use of STECA-Priv in early life cycle stages does not [1] S. Shapiro, "Privacy Risk Analysis Based on System Control obviate the need for downstream risk analysis, such as Structures: Adapting System-Theoretic Process Analysis for Privacy STPA-Priv, once the system is more fully fleshed out; earlier Engineering," 2016 IEEE Security and Privacy Workshops (SPW), assumptions and inferences may no longer be valid and/or San Jose, CA, pp. 17-24, 2016. other privacy risks may have been introduced. We are [2] N. G. Leveson, Engineering a Safer World: Systems Thinking currently applying STECA-Priv to a real-world identity Applied to Safety. Cambridge, MA: MIT Press, 2011. management project, which should provide insights into its [3] W. Young and N. G. Leveson, "An integrated approach to safety and utility and practicality. As it is a new method, modifications security based on systems theory," Commun. ACM, vol. 57, pp. 31- 35, February 2014. based on that experience may be needed, though we are confident that the fundamental soundness of the approach [4] C. H. Fleming, “Safety-driven Early Concept Analysis and Development” (PhD diss., Massachusetts Institute of Technology, will be validated. 2015). Available at http://sunnyday.mit.edu/Fleming-dissertation- final.pdf. Clearly, we also believe in the potential value of [5] Model-Based Systems Engineering (MBSE) Wiki, combining STECA-Priv with MBSE. The ability to model a http://www.omgwiki.org/MBSE/doku.php, accessed February 8, system’s inferred privacy control structure and to execute 2017. that model with integrated constraints adds further rigor to [6] D. Nichols and C. Lin, “Integrated Model-Centric Engineering: The the application of STECA-Priv. It supports identification and Application of MBSE at JPL Through the Life Cycle,” INCOSE correction of privacy control problems early in the SELC. International MBSE Workshop, Los Angeles, CA, January 26, 2014. While STECA is premised on the availability of a system Available at http://www.omgwiki.org/MBSE/lib/exe/fetch.php? ConOps, this is arguably an arbitrary target. Whether media=mbse:06-iw14-mbse_workshop-application_of_mbse_at_jpl_ sufficient descriptive information exists to justify the effort through_the_lifecycle-nichols-lin-final.pdf. required to apply STECA-Priv will be a case-by-case [7] A. W. Wymore, Model-Based Systems Engineering: An Introduction judgment call. to the Mathematical Theory of Discrete Systems and the Tricotyledon Theory of System Design, Boca Raton, FL: CRC Press, 1993. So too will be the use of MBSE in conjunction with [8] M. R. Calo, “The boundaries of privacy harm,” Indiana Law Journal, STECA-Priv. Our experience with a representative tool has vol. 86, pp. 1131-1162, 2011. been that the learning curve is steep. Even once that learning [9] A. van Lamsweerde, Requirements Engineering: From System Goals curve has been climbed, developing an executable SysML to UML Models to Software Specifications. Chichester, UK: Wiley, 2009. model of a complex system will require a substantial [10] J. C. Nwokeji, T. Clark and B. S. Barn, "Towards a comprehensive resource commitment, more so if the investment is to be Meta-Model for KAOS," 2013 3rd International Workshop on Model- maintained over time by capturing all future system changes Driven Requirements Engineering (MoDRE), Rio de Janeiro, 2013, in the model. However, we believe that the potential value of pp. 30-39. such models may be significant, enabling the detection of [11] F. Jaime, A. Maña, Z. Ma, C. Wagner, D. Hovie and M. Bossuet, privacy risks, including emergent system properties, initially "Building a privacy accountable surveillance system," 2015 3rd and as the system undergoes changes. While our illustrative International Conference on Model-Driven Engineering and Software Development (MODELSWARD), Angers, 2015, pp. 646-654. example is relatively straightforward, real-world complex [12] A. Kung, C. Jouvray and F. Coudert, "SALT frameworks to tackle systems are decidedly less so. We anticipate MBSE typically surveillance and privacy concerns," 2015 3rd International would reveal unexpected privacy control problems. This Conference on Model-Driven Engineering and Software Development would be especially valuable (and worth the resource (MODELSWARD), Angers, 2015, pp. 665-673. expenditure) for systems that are intrinsically high risk due to [13] T. Weilkiens, A. Scheithauer, M. Di Maio and N. Klusmann, their data, technology, and/or usage contexts. "Evaluating and comparing MBSE methodologies for practitioners," 2016 IEEE International Symposium on Systems Engineering (ISSE), Further work is needed to develop criteria for when an Edinburgh, 2016, pp. 1-8. explicit constraint in the form of a constraint block is or isn’t [14] S. Shapiro, “Degrees of Freedom: The Interaction of Standards of needed. In the case of the former, it would also be desirable Practice and Engineering Judgment,” Science, Technology, & Human to develop guidance on how to formulate such constraints Values, vol. 22, no. 3, 1997, pp. 286–316. based on the control models as represented in SysML. That