Public Policy Challenges: An RE Perspective David Callele Birgit Penzenstadler Krzysztof Wnuk Dept. of Computer Science Dept. of Comp. Eng. and Comp. Sci. Department of Software Engineering University of Saskatchewan, Canada CSULB, Long Beach, USA Blekinge Institute of Technology Callele@cs.usask.ca birgit.penzenstadler@csulb.edu Karlskrona, Sweden Krzysztof.Wnuk@bth.se Abstract— In this perspective paper, we investigate the paral- Given the perceived correspondence between the domains, lels between public policy and IT projects from the perspective of we decided to investigate further. However, we are not public traditional RE practice. Using the mainstream media as an in- policy experts and we chose to investigate the issues just as formation source (as would an average citizen), over a period of members of the public would do, using the information source approximately one year we captured documents that presented most readily available – the Main Stream Media (MSM), rather analyses of public policy issues. The documents were categorized into eight topic areas, then analyzed to identify patterns that RE than using the (traditional) peer-reviewed literature. In other practitioners would recognize. We found evidence of policy fail- words, we wanted to know whether public policy initiatives ures that parallel project failures traceable to requirements engi- that received MSM coverage appeared to have any characteris- neering problems. Our analysis revealed evidence of bias across tics revealed in their reporting that confirmed the analogy with all stakeholder groups, similar to the rise of the “beliefs over RE for software artifacts. We observed evidence of bias in the facts” phenomenon often associated with “fake news”. We also reported positions, bias in those doing the reporting and even found substantial evidence of unintended consequences due to evidence of “fake news” effects. inadequate problem scoping, terminology definition, domain Our initial investigations led to the following research ques- knowledge, and stakeholder identification and engagement. Fur- ther, ideological motivations were found to affect constraint defi- tions: nitions resulting in solution spaces that may approach locally 1. Can we identify challenges associated with defining, for- optimal but may not be globally optimal. Public policy addresses mulating and realizing public policies? societal issues; our analysis supports our conclusion that RE 1.1. Do the challenges have analogs in RE for software in- techniques could be utilized to support policy creation and im- tensive systems? plementation. (Abstract) 2. How could RE techniques help mitigate the identified pub- Index Terms—Requirements engineering, public policy, bias, lic policy challenges? unintended consequences, mainstream media, ideology and belief, 2.1. Can RE techniques be used to proactively identify failure. (key words) possible public policy challenges during formulation and before enactment? I. INTRODUCTION To answer these questions, we performed an explorative We believe that there is a strong parallel between crafting case study using North American mainstream media and cate- public policy in response to (societal needs to meet) citizen’s gorized the motivating problem, goals and solutions for eight goals and software crafted (in response to requirements) to topics that received significant MSM coverage over the study meet stakeholder goals. In this context, we define public policy period. The study materials were gathered by monitoring news as the mechanism through which societal challenges are identi- feeds (e.g. Google News) for a period of approximately one fied and addressed by the creation of policies, laws and regula- year and capturing those documents that presented a public tions as enacted by government. We see sustainability as a sig- policy issue along with analysis or commentary. We reviewed nificant societal challenge that could be addressed by effective the documents en masse, then categorized and coded them. policy creation and implementation. Our analysis revealed evidence of (apparently unintentional Requirements Engineering (RE) practices such as goal and often large-scale) side effects. These unintended artifacts identification and modeling, requirements analysis, require- appear to exhibit many of the classic RE problems that occur ments negotiation, prioritization and triage have direct corre- during the development of software-intensive systems. spondence with the political process of policy identification, The rest of this paper is organized as follows. In Section 2 policy creation and with resolving challenges associated with we review prior and related work. Section 3 presents the re- realizing policy goals [16]. What is not as clear is the corre- search methodology, research design and discusses threats to spondence between RE practices associated with identifying validity. Section 4 describes the data collection and analysis risks, threats and unintended consequences [11], and develop- efforts and Section 5 presents our observations. A supplemen- ing appropriate mitigation strategies for the proposed policies. tary discussion follows in Section 6 and Section 7 presents the Unintended consequences and mitigation strategies are particu- conclusions and directions for future work. larly important for sustainability initiatives. Copyright held by the author(s). II. PRIOR AND RELATED WORK Wright and Hinson [54] analyze the impact of social media We present related work from the topic areas of ideological on public relations practices and conclude that traditional news biases in stakeholders, mainstream media as information source media still receive higher credibility than social media. in RE, challenges of data mining versus humanism, and prob- Maalej [30] and Pagano [33] have used app store reviews to lem analysis in other domains using RE tools. extract requirements. Guzman and Maalei [19] found sentiment analysis to be very insightful. App store reviews are signifi- A. Ideological biases in stakeholders cantly different from the mainstream media analyzed in this The works on ideological biases in stakeholders are princi- paper, but also use public opinions for informing RE practice. pally in the area of business policy. For example, in 1986, Guzman and Maalei also investigated Twitter messages to Shrivastava [45] discusses whether strategic management is understand their potential to help requirements engineers better ideological, and reviews 20 years of strategic management and understand user needs, using the micro-blogging system as an business policy research and practice. He points out critical additional information source for RE. In contrast, our research criteria like the denial of contradiction and conflicts as well as uses RE analysis to understand parallels between RE for soft- the naturalization of the status quo, and advocates for an open ware intensive systems and crafting public policy. conversation between managerial interests and societal stake- C. Challenges of data mining versus humanism holders of organizations. Parts of such an open conversation, albeit very limited, are mass media articles like the ones ana- Manovich [31] discusses the promises and challenges of big lyzed in the current work. social data with the optimistic conclusion that the new, en- Handelmann et al. [25] discuss ideological framing in larged surface and enlarged depth could facilitate asking new stakeholder marketing based on a longitudinal analysis of types of research questions. stakeholder dynamics in more than 2,000 articles from 45 years Kirschenbaum [28] explores the opportunity of using data of grocery retail trade. They conclude that the interpenetration mining for literary criticism in digital humanities. Kirschen- of strategic and institutional factors has implications for stake- baum rightfully argues that literary criticism rarely uses ground holder marketing. This ideological influence on institutions is truth, and that data mining could point out outliers that ‘pro- also detectable in the media analyzed in our study. voke’ human subject experts. The authors conclude that “While Entine [16] critiques the myth of social investing based on there will hopefully always be a place for long, leisurely hours an analysis of the flaws of proclaimed objective ratings and of spent reading under a tree, this is not the only kind of reading ‘socially responsible’ businesses and their strategies. He con- that is meaningful or necessary.” (p. 5) [28] This result may cludes that social screening is highly anachronistic and based indicate that the current work may be observing some, or all, of on ideologically constructed notions of corporate social respon- the same characteristics. sibility. Taking a stance against Entine’s analysis, Waddock Sculley and Pasanek [42] investigate the impact of implicit [52] explores the myths and realities of social investing and assumptions in data mining for the humanities and argue that provides evidence of the objectiveness of the ratings while not- the standards for evidence production in digital humanities ing their remaining issues. We see similar tendencies of cri- should be even more rigorous than in empirical sciences. Their tique and counter-critique in some of the news articles we ana- most important conclusion is to keep the “boundary between lyzed – two sides with reasonable arguments, and the use of computational results and subsequent interpretation as clearly inflammatory terms elicits stronger responses from the public. delineated as possible.” B. Mainstream Media as Information Source in RE D. Problem analysis in other domains using RE tools Chomsky [14] discusses what makes mainstream media Chandrasekaran [12] provides a task analysis of design “mainstream”. He elaborates that most of mass media is in- problem solving. Byrd et al. [8] synthesize research on re- tended to divert attention (consumers as spectators), the elite quirements analysis and knowledge acquisition techniques for media is geared towards the educated, wealthy and powerful, management information systems. and most academic articles are still within the boundaries of The requirements engineering community has made signifi- institutional obedience. He concludes that, from these charac- cant contributions in the area of legislative work, for traceabil- teristics, we can predict what we would expect to find in the ity and analysis [2][6], for resolving cross-references [38], for current work – and we did. conformance checks [2], and for technology transfer [39]. Kwak et al. [29] compare user-generated content to main- There is further work in the legislative application domains of stream-media-generated content, specifically around sport public governance [1], taxes [46], medical device development communication, and concludes that message valence had a [27], procurement [40][41], nuclear [50], aviation [49], auto- strong impact on triggering biased source evaluation and atti- motive [29], and corporate intellectual policy [9]. The work at tude. We see a similar tendency in the streams we analyzed. hand expands this body of work to new areas. Newman [32] explores mainstream media and the distribu- Due to space restrictions, there are large areas of work tion of news. He highlights the contribution of social media to within RE which this work has not referenced. social discovery and their function as network nodes for social III. RESEARCH METHODOLOGY distribution – and points out the disruptive effects this has on the business models of news organizations. We conducted an exploratory case study over a period of approximately one year during which we investigated public policy topics where there was significant Main Stream Media • Climate change (e.g. resilience, carbon emissions, en- (MSM) press coverage. The MSM was used as an information ergy, electric vehicles, pipelines) source, rather than the academic literature, because we were • Controlled substances (e.g. state versus federal law, focused upon public policy and the MSM is the principal in- avoiding crime, licensing, taxes) formation source for members of the general public. • Equalization (e.g. income, taxes, resources, cost of liv- The MSM was monitored using news feeds such as Google ing) News (https://news.google.com), content aggregators that can During the coding phase, we attempted to identify the chal- be trained (via click through on articles) to perform some de- lenges that the policies were meant to address and the subse- gree of filtering upon the vast quantity of available material. quent problems that arose because of the policy implementa- Whenever we identified an article related in some way to an- tion. We then mapped the results to traditional RE nomencla- nounced public policy and the author’s commentary identified ture (e.g. in some articles we found indicators of inadequate inadequate results or unintended consequences, we then cap- stakeholder identification). A sample of the coding sheet is tured that article to the document repository for later analysis. presented in Table 1. The resulting dataset is a collection of 152 articles or doc- uments on government policies, policy topics or policy initia- A. Threats to Validity tives, government procurement and policy implementation This study has several validity threats that need to be dis- strategies. Sustainability was the primary focus of 37 of the cussed. One of the significant construct validity threats is the articles or documents. In each case, the topic of the article was assumption that RE processes and policy crafting processes an initiative that was (seemingly) made with the best of inten- share a strong parallel. We believe that the collected evidence tions. Unfortunately, the results ranged from simply inadequate and discussion presented in the paper provides sufficient evi- to outright failure and the incidence of (potentially large-scale) dence to support our claims. Still, further empirical validation unintended consequences was high. We include in the category of this assumption needs to be performed in the future. of unintended consequences, policies that even a superficial RE The most significant threat to internal validity is that the analysis would identify as probably not achievable given the observed unintentional effects and consequences have not been solution constraints. The unintended consequences were either statistically analyzed or confirmed. We have not used experi- explicitly identified by the author of the article or they were mental methods to study the effect of changes in the independ- identified after our own analytic efforts (e.g. diverging or con- ent variables on the dependent variables (for example, involv- tradicting policy goals) or prior domain experience. ing a class of stakeholders in relation to unintended conse- As a counterpoint to the MSM sources, we also investigat- quences). However, the study has an exploratory nature and we ed sustainability policies in California, USA [9][36]. We had do not claim that the presented consequences are complete or access to very detailed policy and implementation plans that true for all contexts. had large investments in their development and which we ex- Conclusion validity threats have limited impact on this pected to be of significantly higher quality than the MSM doc- study since we have not used statistical tests to obtain our re- uments and to be relatively bias-free. sults. At the same time, We made several efforts to minimize At the end of the document collection phase, the documents subjectivity and resolve potential conflicts when analyzing and were reviewed in their entirety in two sessions totaling approx- categorizing qualitative evidence. We used researcher triangu- imately 12 hours. We used researcher triangulation to decrease lation to decrease the subjectivity bias, with the first two au- the subjectivity bias, with the first two authors performing the thors performing the analysis in discourse and the third author analysis in discourse and the third author reviewing the coding reviewing the coding and interpretation for consistency and and interpretation for consistency and correctness. The coding correctness. was emergent and led to the following eight categories. Given Since the study is exploratory, external validity remains the the topic areas, there is some potential that a document could main limitation of our work. We aim for analytical generaliza- be coded into more than one category; the final placement was tion rather than statistical generalization [16] and present the based on discussion among the researchers. case and method details to enable replications and further stud- • Algorithms (e.g. big data analysis, artificial intelli- ies. Still, we studied only a limited dataset of 152 articles on gence) that have drawn sufficient attention to warrant government policies and policy initiatives. public policy discussions We note that we are taking a humanistic approach to our • IT projects (e.g. large-scale publicly funded projects, analysis. While there is a significant body of research in generally in support of some policy goal) automated processing of news feeds and sources like Twitter, • Social (e.g. free speech, critical thinking, gender issues, that work generally analyzes large corpuses. Unfortunately, fake news, radicalism) that is not the way “the average person works”; they do not • Privacy (e.g. location data, social media, children’s read hundreds or thousands of articles on an issue, they might self-determination) read one or two. This is a substantial validity threat, but we • Policy (e.g. cybersecurity, copyright, taxes, housing) mitigated this risk by individually reading every article and performing the final coding after discussion. TABLE 1. DETAILED CODING SAMPLE ID Category Goal Solution Unintended Keywords Problems Consequences Reuters AI Algorithms Make news faster, Use 13 AI algorithms Potential to eliminate the News, media, Stakeholders more accurate, and that mine Twitter and jobs of 2500 highly AI, algorithm not considered more resilient against cluster and extract educated and skilled fake news attempts. news reporters Amazon sales Algorithms Make customers want Sales algorithm that (1) Suggesting composites Sales, online, Unintended algorithm to buy more and feel works well in selling that are potentially useful algorithm consequences, well taken care of things that are useful to build explosive devices, incorrect because frequently (2) could potentially serve interferences bought together to detect potential terrorists Passport Canada IT projects / Make passports safe Electronic system to At least $75 million over Passport, IT Failed to New passport and secure with produce the new budget and well behind consider processing physical robustness passports schedule. The project "did specific quality system not include a plan for requirements security requirements." Sight of personal Privacy Software company Software can be given Loss of privacy, complete - Privacy EU has data privacy, Google, needs to make money away for free if we unintended by the user. privacy policies Facebook collect the users’ data Unintended consequence that would make instead by the companies is a lot of US regulatory pushback of startups illegal various jurisdictions. No understanding of consequences (e.g. why I didn't get the job/loan/etc.) Controlled Controlled Get people to relax and Pot legalized. Potentially reversing Controlled A federal substances substances not be anxious. policy. You smoke pot in substances marshal could summary Cali, leave to EU, return, if walk across the you deny to have smoked US and throw upon return to the US you anyone in jail can be charged, perhaps that has smoked jailed. pot. Equalization Equalization Ensure that all political A formula. It is People don't have trust that Equalization People don't summary regions in the country criticized as being far equalization is fair. Some bother to learn – are able to provide too complex. From a regions experience greater what it costs to approximately mathematician's levels of taxation than deliver a public equivalent public perspective the model others. The people who policy initiative. services to the citizens is grossly inadequate, receive the money say Stakeholders inaccurate, and everyone gets taxed the can’t understand simplistic. First year same, but the federal a correct calculus is way more government takes that solution so they complex than this money and reallocates it as (grudgingly) model, but the general rebates to different regions, accept a flawed public still perceives it so it has the same effect as solution. as too complex. different taxation. California California California aspires to be How many things are Millions of dollars are Sustainability Prioritization Sustainability Sustainability a thought-leader and being considered at being spent on these policy Policies summary Policies puts into action a lot of CSULB and the Port initiatives. What could what we have learned of LA as examples - have been achieved if that on mitigating shows you can make money would have been sustainability substantial differences spent somewhere else, e.g. challenges over the but at high costs in India or another past years. developing country where it can help far more people with the same resources? of the categories Controlled Substances, Equalization, and Cal- ifornia Sustainability Policy, as we found the results more in- IV. REFLECTIONS UPON THE METHODOLOGY sightful on the aggregated level. For each row, we identify the We retrieved and analyzed 152 articles and an excerpt of Goal as the original intention for the policy and the Solution our analysis is presented in Table 1. The left column indicates that was chosen. We then identify the Unintended Consequence the identifier of the news item, then the category into which we arising from that solution. We further tagged with Keywords classified the article. The bottom three rows are summary rows and identified Problems of the scenario. For example, the first row identifies the issue of the Reuters V. OBSERVATIONS AI, in the category Algorithms, where the decision makers had We begin each grouping of our results with a descriptive the intention of making news faster, more accurate, and more label, present our observations and, generally, present one or resilient against fake news attempts (in response to public out- more (sometimes rhetorical) questions. cry and nascent public policy initiatives). The established solu- Legislative Contradictions: We observed cases where tion was to deploy 13 AI algorithms that mine Twitter feeds to there are contradictions within legislation – how do the indi- identify topics of interest. The (potentially) unintended conse- viduals responsible create legislation that contradicts? Are quence of the desire to more quickly react to current events is these conflicts deliberately created by those responsible or is that the jobs of 2,500 highly educated and skilled reporters are there something else influencing (and possibly corrupting) the potentially being eliminated. We associated the keywords process? For example, on the topic of marijuana legalization, news, media, AI and algorithms, and the main problem that all individual states in the USA have decriminalized personal use relevant stakeholders were not considered. while federal law continues to make possession a crime. Legis- The retrieved articles are dominated by works wherein the lators in individual states have deliberately chosen to contradict author reflects upon some policy initiative and the associated federal law. Would we tolerate conflicting, and unresolved, successes and failures. These kinds of articles appear to be in- requirements when designing a software intensive system? herently biased towards negative critique (perhaps in an effort Related work represented policy constraints as logic program to generate more page views?) and they became a rich source, [44], but that is only a very first step in solving these issues. perhaps even a treasure trove, of unintended consequences. Same Old Problems: Despite 50 years of experience in IT These articles all contain strong observer bias (they are opinion systems, the last 30 years of (approximately) which RE has pieces), but we use them as data sources anyway – for these are been a formal discipline, we observe that system after system the same data sources that shape their reader’s opinions and continues to experience problems such as missing requirements perceptions. After all, just because a source is biased it does not and missing stakeholders. For example, the Government of mean that the inherent message is not reality to the reader. We Canada embarked upon the creation of a unified payroll system also gathered resources for two instances where policy was for all federal government employees. The system must man- reduced to practice (sustainability and environmental policy age the contracts for hundreds of different unions, each of initiatives at California State University Long Beach and the which has their own pay scales, promotions, benefits packages Port of Long Beach, including significant traditional engineer- and retirement plans. Individual employees could spend their ing technology analyses and engineering economic analyses). entire career within a single union or change to a new union The first two authors coded the articles in discussion and each time they change the position in which they are employed. we applied significant domain knowledge of their own to pro- RE practitioners would immediately recognize the likelihood of vide context for the observations and to enhance the richness of a combinatorial explosion in the business rules and data ele- the conclusions. This technique has the potential to provide ments that must be managed and would identify the issue to the greater insight but is also a significant threat to validity. We are stakeholders. In this case, the issue appears to have been trivial- trying to be humanistic in this work, we are analytical but not ized or ignored, and while we do not have any “insider infor- coldly so. In other words, we are reacting as people, not as a mation” that would allow us to elaborate further, we do note machine algorithm. We are observing emotional content and that the project is considered a near-complete disaster by all there is the potential that we have introduced some of our own stakeholders and projected implementation and remediation emotional bias on some of the topics. For each article, we read costs are in excess of 400% of the original budget (the Gov- the content and (typically) the first 50 to 100 reader comments ernment recently announced1 that cost estimates have exceeded (assuming that comments are present). $1B CDN and the creation of a task force to find a replacement Within the first few months of our study, we realized that before this system is even fully functional). Other government our data set would have an inherent bias: it would not be unrea- IT projects (especially those related to health care) do not seem sonable to assert that the MSM generally reports upon things in to fare much better. a negative manner, and the associated comments are often Holistic Perspectives: What is possible and highly desired more extreme than the studied article. from a political perspective is often not possible from an eco- With this realization we adjusted our research effort, focus- nomic perspective and it seems that policy makers rarely take ing more of our efforts upon those reports wherein there ap- this holistic view. For example, promoting the use of electric peared to be unintended consequences of some public policy cars should reduce CO2 emissions and is relatively easy to jus- initiative. We note that there were many, many reports of unin- tify if the only metric used is emissions per distance traveled. tended consequences and not all were negative. We then re- However, electric vehicle production creates significant CO2 fined our effort to identifying the unintended consequences and emissions [17] and the consumption of significant quantities of evaluated them using a system model. Our analysis was based rare metals. The electricity used must be generated by low on the question “If this was a software system and we were emission sources and (somehow) delivered to the vehicles. It is performing a post hoc evaluation using RE techniques, what well-known in electrical engineering practice that the North observations and recommendations would we make?” The complete codebook is available on Google Drive [15] 1 http://www.ctvnews.ca/politics/minister-fixing-phoenix-pay-system-could- cost-1b-1.3672663 American electric distribution grid was not designed to deliver address these differences. Despite the well-established princi- this much energy and can only hope to do so with careful de- ple that federal powers overrule provincial powers, individual mand management (e.g. only charge your vehicle after 9 p.m. if provinces that do not agree with federal policy on a given topic you live in a residential neighborhood) unless significant in- can, and do, attempt to override the federal policy by crafting vestments are made in infrastructure improvements. Despite confounding or competing legislation within those aspects of the challenges, to achieve the desired public policy objectives exclusive provincial jurisdiction. For example, inter-provincial systemic changes and early adopters will be necessary. For this, and international pipelines are clearly placed under federal ju- we can use the foundations of systems thinking [13][21] and risdiction. Environmental regulatory powers exist at both the apply them to engineering activities [3]. federal and provincial levels and anti-pipeline activists attempt Identifying the “Right” problem: Our review identified to use provincial environmental regulatory powers to impede or numerous cases where those directly involved with a policy or completely block any federal approvals for such projects. project appear to believe that they have correctly identified the While conflict identification and resolution have been targeted problem, and that their proposed (or actual) solution addresses by [51][4], these policy-level conflicts require a more holistic the problem. However, when other parties look at the problem level of modeling and reasoning than technical requirements. they quite strongly disagree upon the problem definition – con- Privacy: Even though users legally agree to having their sider the acrimony that exists between perspectives on climate data tracked by many of the apps and services that they use, change challenges and proposed solutions. This pattern implies most people grossly underestimate the magnitude of the data that there is a class of problems where perspective is very im- trail that they create. While this data collection activity is pre- portant. If that is the case, is our established body of RE prac- sented to the user as a way to improve the user experience, tices applicable to those problems? Does RE have to evolve to more and more users are learning that the same data can also be able to support these problems or do we just ignore that have significant unintended consequences – especially when class of problems? This concern is partly addressed by some that data is licensed to a third party. We routinely see reports of work in RE on viewpoints[48], but only on a level of technical individuals that use social media experiencing negative conse- representation in requirements documentation. Do these prob- quences (e.g. denied insurance claims, denied bank financing, lems also affect RE for (software-intensive and other) systems? inability to get job interviews, etc.) [26]. Side Effects: When reviewing the articles, we were repeat- Public policy is responding to these reports, most notably in edly given the impression that comprehensive analyses of po- the European Union, and there is increasing pressure on the tential complicating factors is either performed badly or not providers of these services to support correction and deletion of performed. This is an area where RE can significantly contrib- data collected about individual users, including the “right to be ute beyond the work in [55]. For example, the Swedish gov- forgotten” [19] [53]. But, what about the effect that the data ernment performed an analysis that showed that (in their opin- had on the analyses before it was modified or deleted? And ion) too many motor vehicle accidents occurred when overtak- how does the modification or deletion request propagate to ing (passing). To reduce the accident levels, flexible posts were third parties that may have a copy of the original data or anal- installed in numerous stretches of the roads. While these flexi- yses that were based upon the original data? How do we con- ble dividers may have reduced the accident rate for cars and struct requirements not just for the originating system but also trucks, they have made travel more dangerous for motorcycle for third-parties? riders who cannot hit these barriers without serious conse- Biases in AI and Data Mining: Bias in automated deci- quences. sion-making systems is receiving ever-increasing public policy (Magnitude of) Unintended Consequences: The unin- attention. Closely related to privacy issues, deliberate or unin- tended consequences of the policies under investigation have a tentional bias has the potential for significant unintended con- much larger scope and scale than we expected. And, larger than sequences. The specialists in the field don’t always know why the original policy intervention necessarily would have made they are accepting the results they get out of the algorithms, many people believe. leading to backlash from observers (“Will anyone ever write As an example, consider affirmative action policies whose another positive story about a tech startup? I said probably not” goals are to “level the playing field” between disparate groups. [22]). If the algorithms that are being used to mine these data Superficially, these policies obtain at least grudging acceptance repositories have biases (intended or unintended), they may by a majority of the populace in North American jurisdictions. amplify negative conclusions about individuals that are un- However, we see evidence that different special interest groups founded or unwarranted. The same technologies can also be “weaponize” these policies, in different ways, and use them to applied to induce bias in users, from addictive video game increase conflict rather than decrease conflict. As a result, posi- properties and Facebook’s deliberate design to induce emotion- tions become ever more polarized and compromise solutions al reward to the numerous reports of election interference in the become more difficult to achieve. US presidential elections2 and the Brexit campaign. Affected Domains: Different regions within a state, prov- Significant elements of the technology sector could find ince or country tend to have different social mores and these themselves regulated, or at least required to justify or defend can translate to differences in local legislation and increased potential for conflicts. In Canada, legislative powers are delib- 2 erately split between the federal and provincial levels to help to https://www.theguardian.com/technology/2017/oct/30/facebook-russia-fake- accounts-126-million their algorithms in ways that can be comprehended by policy ple simply cannot afford to travel except in absolute necessity makers and by the general public. What happens when the will have the effect of reducing emissions, but is this even pos- technology sector answers “We don’t know. We just know that sible to implement in a democratic society where people need it (seems to) work better than anything that we have done be- to travel to work? If you believe that climate change is caused fore.”? What will happen to “technological progress” if every- by human intervention, perhaps you could target non-essential one believes that they have “the right” to provide input any travel – for example, ban tourism. Superficially, this would time that they believe that this class of algorithms is affecting create non-trivial reduction in emissions. However, such a poli- their lives (or they will threaten to claim some form of oppres- cy would eliminate a significant source of income for many sion or human rights violation)? How many companies that developing nations and seriously impede their ability to offer rely upon data mining would find their business models at risk public services such as health care. in such a regulated environment? This is a significant unintended consequence. The policy Deployment of these technologies in support of public poli- would destroy the livelihoods of everyone in the tourism indus- cy will require us to truly understand what is “going on inside.” try and of many third world nation service industries – is that Otherwise, how do we evaluate whether we are or not exclud- what the ‘environmentalists’ want to happen? We posit that this ing people from fair treatment in our society based upon what is unlikely. some algorithm returns as a result. In the world of the movie To make this point even more strongly, the Government of “Minority Report”, precognition was combined with significant Canada attended the Paris climate convention and signed the technical support to eliminate murder. What happens when we Paris climate accords. Later that year, the Parliamentary Budget replace precognition with data mining and AI? The computa- Office (an agency that provides independent cost analyses of tional techniques may be mathematically accurate but how do parliamentary initiatives) issued a report that sought to bring we know if we are correctly interpreting the results? If, for ex- the commitments into perspective for the average citizen ample, ethnic heritage in combination with neighborhood, so- [23][34][46]. The report identified that achieving Canada’s cioeconomic status and educational level, leads to a person commitments would require emissions reductions of a magni- being identified as a potential future criminal, does that mean tude that was more than the equivalent of the elimination of all the individual is indeed a criminal and should be treated as motorized transportation in Canada – no aircraft, no shipping, one? Does that mean that the analysis has identified fundamen- no busses, no cars, no motorcycles, etc.. How can a govern- tal flaws or failures in society’s structure? Even though these ment maintain any credibility with its citizens if they make factors often correlate, they are not necessarily causal and, commitments that appear to be unachievable? After all, mobili- therefore, should we be working on fixing the cause of the ty of people and goods lies at the heart of the global economy problem and not the symptom? Finally, if the models are telling and while the government’s actions were strongly supported by us things we don’t want to hear, then maybe the models are the environmental movement, the average person’s position has simply identifying opportunities for improvement. shifted toward disbelief, apathy and resentment. Rather than Social Perspectives: We identified a pattern of hardening fostering support for the initiative, they have created resistance. of positions by factions interested in public policy topics. Ra- In contrast, sustainability initiatives in California underwent ther than looking for compromise, it appears that the factions significant planning efforts, culminating in realistic implemen- are treating issues as a zero-sum game: “we adopt my position, tation plans [9][36]. Even though engineering economic anal- or else…” Can RE techniques (especially those related to con- yses showed that some of the goals were not cost effective, an flict resolution and mediation) be used to find common ground informed decision was made to proceed in pursuit of those between these stakeholders? What does it mean to practitioners goals – unlike the public perception of the Canadian initiative. (and society in general) when stakeholders tell us there can be Emotional Content: Emotionally charged content is preva- no validity in a common ground? As Brown points out, in our lent across many of the articles, as evidenced by the author’s current society there is a “phenomenon of you are either with selection of adjectives and adverbs and by the positions taken us or against us.” [7] This behavior pattern, if it continues to by supporters and detractors within the accompanying com- grow, is serious cause for concern. ment sections. From the Twitter storms of President Trump and For example, this pattern is very evident in people’s posi- his interactions with Kim Jong-un to people issuing threats on tions about climate change. Do you believe humans contribute social media platforms toward people who oppose their posi- to something that is called climate change? Do you believe that tion on issues of the day, how do we get past all of that negativ- greenhouse gasses can be absorbed by the environment without ity and unwillingness to compromise to even get to the point of significant damage or not? Do you feel we should be minimiz- being able to agree upon a goal, let alone solutions? Are these ing our human byproducts and pollution? behavior patterns evident even when performing RE for soft- You can interpret these questions with sufficient qualifiers ware intensive systems? such that eventually you will get almost every climate change Time: The time needed to introduce and pass legislation in denier or promoter to agree. For example, many climate change support of policy initiatives (e.g. reduce industrial CO2 emis- deniers are not against mitigation policies per se, rather they sions), and to see the effects of the policies (often measured in tend to be against specific policies because they do not believe decades), is much longer than the average time a government that those policies are a cost-effective solution to the problem. holds power (four to six years in most democratic countries). For example, taxing carbon emissions at such a level that peo- This reality has led to two patterns: New governments try to reverse policies set by previous governments resulting in abort- would be more organized and better structured than they are, ed efforts and significant waste and, in anticipation, current but we can’t help but wonder what it would be like if they were governments try to establish policies in such a way that they as relatively error-free as well-crafted software. cannot be easily modified. What is lost if the original policy implementation was actually going to be effective? How does a VII. CONCLUSIONS AND FUTURE WORK new government undo a policy that has proven deleterious? In this work, we investigated the use of the mainstream me- dia as a data source for an analysis of public policy and related VI. DISCUSSION governmental initiatives. Our first research question asked Using the MSM as a data source, rather than peer-reviewed whether we could identify challenges associated with defining, academic papers is certainly a ‘different’ research experience. formulating and realizing public policies from this data source. The prevalence of opinion, often strong opinion, without sub- We were able to identify challenges by analyzing the content, stantiating evidence converts the quality evaluation process to removing the commentary, then identifying the underlying one of (1) how well is the article written (for me)? (2) does the “facts”. We were also able to identify how people perceived the article align with my personal biases and (3) what do I perceive challenges, which often were biased by ideology. Furthermore, to be the reputation of the author? When one includes the we found that the challenges do have analogs in RE for soft- comments in the analysis, it is easy to be affected by the ware intensive systems, so there is potential that RE can help to strength of the (often negative) positions held by the comment- proactively identify possible public policy challenges. ers. This negativity bias can easily be passed on to the analysts A metaphor that we find useful is that public policy is the and this contagion is a known psychological phenomenon algorithm for governing the operation of the machine that is [42].3 Researchers interested in performing a humanistic inves- society. As such it is easy to answer the next research question tigation into these materials are advised to be prepared for the in the affirmative: how could RE techniques help mitigate the potential emotional side-effects. identified public policy challenges? We saw no evidence that How does the MSM affect stakeholder perceptions, opin- RE techniques could not be successfully applied in this do- ions, and the hardening of both? There seems to be an accelera- main. Finally, we asked whether RE techniques could be used tion and hardening of positions in mainstream media – is this to proactively identify possible public policy challenges during something that RE might have to consider or be more cogni- formulation and before enactment? The final research question zant of moving forward? For example, do you send your (polit- is not as easy to answer. Certainly, validation and verification ically) left-leaning team in when you have a (politically) left- techniques could be used to identify issues and mitigate risks if leaning client? Such a proactive effort can amplify the biases the participants had sufficient domain expertise and, dare we (prejudices) but has the potential to lower the risk of mis- say it, wisdom. communication. We take special note of the seeming rise in We see sufficient evidence for there to be a role for RE in ideological bias on the part of policy makers, thought leaders helping people at large to understand the technology that seems and the general public. This trend toward the adherence to a to overwhelm them, possible consequences and side effects. position or interpretation independent of rational analysis of the While this work is another piece of evidence of the univer- underlying facts could have far-reaching and unexpected ef- sal nature of problem patterns and critical thinking, it has also fects. While we have observed evidence of ideological bias in delivered significant context for future work. Each of the major policy, we must ask whether this trend will have an effect on points in Section V could readily become a research thrust: RE for software artifacts. For example, will practitioners need • Legislative contradictions to be more diligent in exploring stakeholder statements of their • Same old problems wants, exploring whether or not these wants can be evaluated • Identifying the “right” problem as stated (in the transition from wants to needs during prioriti- • Side effects zation) or whether the statements must be further explored to • Magnitude of unintended consequences identify ideological biases? When attempting to understand the risks and threats arising from this trend we are prompted to ask: • Holistic perspectives How does this knowledge inform us about how they (policy • Affected domains makers, their constituents, and politicians) perceive circum- • Biases in AI and data mining stances and issues; what gains, risks and threats does this offer • Social perspectives to RE practice? In this context, miscommunication challenges • Emotional content can potentially be greater than anticipated. • Time We were somewhat surprised, if not shocked, by the num- Upon reflection, we must also ask whether the focus of this ber of instances of open conflict in regulations and legislation. work is even something with which RE practitioners should be Perhaps we were naïve in assuming that the legal structures concerned. What are, and should be, the bounds of RE? Despite the fact we are pushing the bounds of RE, are we pushing too far into ethics and overreaching? Are we oblivious to the fact 3 Associates of the lead author actually held what could (charitably) that there are many other people already attempting to address be called a mini-intervention with him in an effort to determine what these issues? had caused him to become increasingly negative over the prior six months. Other research questions that are prompted by our experi- [6] T. D. Breaux, M. W. Vail, and A. I. Anton. "Towards regulatory ence, but farther afield from RE include: compliance: Extracting rights and obligations to align • Do opinion article writers become thought leaders? Are requirements with regulations." Requirements Engineering, 14th IEEE International Conference. IEEE, 2006. they good barometers of the populous and their emo- tions? [7] B. Brown, “Braving the wilderness: The quest for true belonging and the courage to stand alone”. 2017. New York, NY: Random • Can we use machine learning across the “wisdom of House the crowd” as a means to validate what the pundits and [8] T. A. Byrd, K. L. Cossick, and R. W. Zmud. "A synthesis of politicians are saying? research on requirements analysis and knowledge acquisition • Can we extract the core content of each document and techniques." MIS quarterly, 1992, pp. 117-138. perform formal semantic analysis to identify the biases [9] California State University, Long Beach. “Strategic Energy in the presentation and to quantify the intensity of the Plan.” California State University, Long Beach. July 5, 2011 bias? We could attempt to identify the ideology of the [10] David Callele, Krzysztof Wnuk. More than requirements: policy authors, reporters and commentators, but sense Applying requirements engineering techniques to the challenge that this is far out of our traditional field. of setting corporate intellectual policy, an experience report. • Are things really “as bad” as the MSM would seem to RELAW 2011: 35-42 want to have us believe? Is it possible to know the rela- [11] V. Cerf, “Unintended Consequences” Communications of the tive scale of the negative elements – were the reported ACM, vol. 61, March 2018. negatives only a small proportion of the overall initia- [12] B. Chandrasekaran, "Design problem solving: A task tives? Were the reported negatives only relatively rare analysis." AI magazine 11.4 (1990): 59 occurrences in the greater context of society? [13] Checkland, Peter. "Systems thinking." Rethinking management This work has proven to be a rich source of research ques- information systems (1999): 45-56. tions and opportunities that provide ample pointers towards [14] N. Chomsky, "What makes mainstream media mainstream." Z future work. First, we want to deepen our analysis using empir- magazine 10.10 (1997): 17-23. ical methods and expand on this exploratory study with a quan- [15] Results Codebook, https://drive.google.com/file/d/14ErlsN titative analysis of evidence for said RE challenges. Second, we OOMj79GnXknWvOxu1UtG3ZUB1_/view?usp=sharing want to use that data for a scenario analysis, applying RE tech- [16] W. N. Dunn, Public policy analysis. Routledge, 2015. niques to explore whether we can use this form of analysis to [17] Dyer, Ezra. ”That Tesla Battery Emissions Study Maming the prevent some of the unintended consequences that played out Round? It’s Bunk.” Popular mechanics, June 2017. in those scenarios. Third, we plan to detail a research agenda of https://www.popularmechanics.com/cars/hybrid- other opportunities outside of software and systems engineer- electric/news/a27039/tesla-battery-emissions-study-fake-news/ ing where requirements engineering techniques could have [18] J. Entine, "The myth of social investing: A critique of its significant positive impact and improvement potential for the practice and consequences for corporate social performance situations under analysis. Finally, we plan to perform an RE research." Organization & Environment 16.3 (2003): 352-368. analysis of the goals of the GDPR, their feasibility, validation [19] EUREKA “The right to be forgotten” and verification techniques, and monitor for evidence of unin- https://eureka.eu.com/gdpr/the-right-to-be-forgotten/ Last visited February 28, 2018 tended consequences. [20] B. Flyvbjerg, "Five misunderstandings about case-study re- REFERENCES search." Qualitative inquiry, vol. 12, 2006, pp. 219-245. [21] Forrester, Jay W. "System dynamics, systems thinking, and soft [1] A. Okhaide, M. Alhaj, D. Amyot, O. Bahy Badreddin, E. OR." System dynamics review 10.2‐3 (1994): 245-256. Braun, N. Cartwright, G. Richards, G. Mussbacher, Creating [22] E. Griffith, “The other tech bubble” WIRED, December 16, Quantitative Goal Models: Governmental Experience, 2017, https://www.wired.com/story/the-other-tech-bubble/ ER 2014: 466-473 Last visited February 28, 2018 [2] C. Arora, M. Sabetzadeh, L. C. Briand, F. Zimmer: Automated [23] G. Lorrie “GOLDSTEIN: Trudeau breaks his promise on carbon Checking of Conformance to Requirements Templates Using taxing” Toronto SUN, December 14, 2017 Natural Language Processing. Trans. Software Eng. 41(10), http://torontosun.com/opinion/columnists/goldstein-trudeau- 2015, pp. 944-968. breaks-his-promise-on-carbon-pricing, [3] Bahill, A. Terry, and Bruce Gissing. "Re-evaluating systems Last visited February 28, 2018 engineering concepts using systems thinking." IEEE [24] E. Guzman, W. Maalej. "How do users like this feature? A fine Transactions on Systems, Man, and Cybernetics, Part C grained sentiment analysis of app reviews." Requirements Engi- (Applications and Reviews) 28.4 (1998): 516-527. neering Conference (RE), 2014 IEEE 22nd International. IEEE, [4] Boehm, Barry, and Hoh In. "Identifying quality-requirement 2014. conflicts." IEEE software 2 (1996): 25-35 [25] J. M. Handelman, P. H. Cunningham, and M. A. Bourassa. "Stakeholder marketing and the organizational field: The role of [5] T. D. Breaux, D. G. Gordon. "Regulatory requirements institutional capital and ideological framing" Journal of Public traceability and analysis using semi-formal Policy & Marketing 29.1 (2010): 27-37. specifications." International working conference on requirements engineering: Foundation for software quality. [26] J. Harris, “Take it from the insiders: Silicon Valley is eating Springer, Berlin, Heidelberg, 2013 your soul” The Guardian, January 1, 2018 [27] H. Dagny Hauksdottir, B. Ritsing, J. Christian Andersen, N. procurement process of IT solutions for the Brazilian Federal Henrik Mortensen: Establishing Reusable Requirements Derived Public Administration” RELAW 2014: 39-40 from Laws and Regulations for Medical Device [42] P. Rozin, E. B. Royzman. "Negativity bias, negativity domi- Development. RELAW 2016: 220-228 nance, and contagion." Personality and social psychology re- [28] M. G. Kirschenbaum, “The remaking of reading: Data mining view, vol. 5, 2001, pp. 296-320. and the digital humanities." The National Science Foundation [43] D. Sculley, M. B. Pasanek. "Meaning and mining: the impact of Symposium on Next Generation of Data Mining and Cyber- implicit assumptions in data mining for the Enabled Discovery for Innovation, Baltimore, MD. 2007. humanities." Literary and Linguistic Computing 23.4 (2008): [29] D. H. Kwak, K. Yu Kyoum Kim, and M. H. Zimmerman. "User- 409-424. versus mainstream-media-generated content: Media source, [44] Sergot, Marek J., et al. "The British Nationality Act as a logic message valence, and team identification and sport consumers’ program." Communications of the ACM 29.5 (1986): 370-386. response." International Journal of Sport Communication, [45] P. Shrivastava, "Is strategic management ideological?" Journal vol. 3, 2010, pp. 402-421. of management 12.3 (1986): 363-377. [30] W. Maalej and N. Hadeer, "Bug report, feature request, or simp- [46] J. Snyder, “Canada still to fall short of Paris target”, Financial ly praise? On automatically classifying app reviews. "Require- Post Canada, December 29 2017, ments Engineering Conference (RE), 2015 IEEE 23rd Interna- http://business.financialpost.com/commodities/energy/canada- tional. IEEE, 2015. still-on-pace-to-fall-short-of-paris-target-despite-ambitious- [31] L. Manovich, "Trending: The promises and the challenges of big climate-policies-un-filings. Last visited February 28, 2018 social data." Debates in the digital humanities, vol. 2, 2011, pp. [47] G. Soltana, M. Sabetzadeh, L. C. Briand, “Model-Based 460-475. Simulation of Legal Requirements: Experience from Tax Policy [32] N. Newman, "Mainstream media and the distribution of news in Simulation” RE2016: 303-312 the age of social discovery." Reuters Institute for the Study of [48] Sommerville, Ian, and Pete Sawyer. "Viewpoints: principles, Journalism, University of Oxford (2011). problems and a practical approach to requirements engineering." [33] D. Pagano and W. Maalej. "User feedback in the appstore: An Annals of software engineering 3.1 (1997): 101-130. empirical study." Requirements Engineering Conference (RE), [49] R. Tawhid, E. Braun, N. Cartwright, M. Alhaj, G. 2013 21st IEEE International. IEEE, 2013. Mussbacher, A. Shamsaei, D. Amyot, S. B. Ahmadi, G. [34] Pilling, David. “Clean electric cars are built on pollution in Richards: “Towards outcome-based regulatory compliance in Congo”, Financial Times, July 2017 aviation security”, RE 2012: 267-272 [35] Canada’s Greenhouse Gas Emissions: Developments, Prospects and Reductions, http://www.pbo- [50] E. J. Uusitalo, M. Raatikainen, M.Ylikangas, T. Männistö, dpb.gc.ca/web/default/files/Documents/Reports/2016/ClimateCh “Experiences from an industry-wide initiative for setting ange/PBO_Climate_Change_EN.pdf metadata for regulatory requirements in the nuclear [36] Ports of Long Beach and Los Angeles. “Roadmap for Moving domain”, RELAW 2014, pp. 2-9. Forward with Zero Emission Technologies at the Ports of Long [51] Van Lamsweerde, Axel, Robert Darimont, and Emmanuel Beach and Los Angeles.” Roadmap for Zero Emissions – Letier. "Managing conflicts in goal-driven requirements Technical Report, August, 2011. engineering." IEEE transactions on Software engineering24.11 [37] Birgit Penzenstadler and Jörg Leuser: Complying with Law for (1998): 908-926. RE in the Automotive Domain. RELAW 2008: 11-15 [52] S. Waddock, "Myths and realities of social [38] N. Sannier, M. Adedjouma, M. Sabetzadeh, L. C. Briand, “An investing." Organization & environment 16.3 (2003): 369-380. automated framework for detection and resolution of cross [53] Wikipedia “The right to be forgotten” references in legal texts”. Requir. Eng. 22(2): 215-237 (2017) https://en.wikipedia.org/wiki/Right_to_be_forgotten, [39] N. Sannier, M. Sabetzadeh, L. C. Briand, “From RELAW Last visited February 28, 2018 Research to Practice: Reflections on an Ongoing Technology [54] D. K. Wright, M. D. Hinson. "An updated look at the impact of Transfer Project”. RELAW 2017: 204-208 social media on public relations practice." Public relations [40] G. Lamartine da Silva Barboza, F. Cysneiros Filho, R. A. C. de Journal, Volume 3, 2009, pp. 68–73. Souza, “Towards Legal Compliance in IT Procurement Planning [55] Yen, John, and W. Amos Tiao. "A systematic tradeoff analysis in Brazil's Federal Public Administration”. RELAW 2016: 229- for conflicting imprecise requirements." Requirements 238 Engineering, 1997., Proceedings of the Third IEEE International [41] G. Lamartine da Silva Barboza, F. Cysneiros Filho, R.A. C. de Symposium on. IEEE, 1997. Souza “Towards a legal compliance verification approach on the .