=Paper=
{{Paper
|id=Vol-2223/paper4
|storemode=property
|title=Public Policy Challenges: An RE Perspective
|pdfUrl=https://ceur-ws.org/Vol-2223/paper4.pdf
|volume=Vol-2223
|authors=David Callele,Birgit Penzenstadler,Krzysztof Wnuk
|dblpUrl=https://dblp.org/rec/conf/re/CallelePW18
}}
==Public Policy Challenges: An RE Perspective==
https://ceur-ws.org/Vol-2223/paper4.pdf
Public Policy Challenges: An RE Perspective
David Callele Birgit Penzenstadler Krzysztof Wnuk
Dept. of Computer Science Dept. of Comp. Eng. and Comp. Sci. Department of Software Engineering
University of Saskatchewan, Canada CSULB, Long Beach, USA Blekinge Institute of Technology
Callele@cs.usask.ca birgit.penzenstadler@csulb.edu Karlskrona, Sweden
Krzysztof.Wnuk@bth.se
Abstract— In this perspective paper, we investigate the paral- Given the perceived correspondence between the domains,
lels between public policy and IT projects from the perspective of we decided to investigate further. However, we are not public
traditional RE practice. Using the mainstream media as an in- policy experts and we chose to investigate the issues just as
formation source (as would an average citizen), over a period of members of the public would do, using the information source
approximately one year we captured documents that presented
most readily available – the Main Stream Media (MSM), rather
analyses of public policy issues. The documents were categorized
into eight topic areas, then analyzed to identify patterns that RE than using the (traditional) peer-reviewed literature. In other
practitioners would recognize. We found evidence of policy fail- words, we wanted to know whether public policy initiatives
ures that parallel project failures traceable to requirements engi- that received MSM coverage appeared to have any characteris-
neering problems. Our analysis revealed evidence of bias across tics revealed in their reporting that confirmed the analogy with
all stakeholder groups, similar to the rise of the “beliefs over RE for software artifacts. We observed evidence of bias in the
facts” phenomenon often associated with “fake news”. We also reported positions, bias in those doing the reporting and even
found substantial evidence of unintended consequences due to evidence of “fake news” effects.
inadequate problem scoping, terminology definition, domain
Our initial investigations led to the following research ques-
knowledge, and stakeholder identification and engagement. Fur-
ther, ideological motivations were found to affect constraint defi- tions:
nitions resulting in solution spaces that may approach locally 1. Can we identify challenges associated with defining, for-
optimal but may not be globally optimal. Public policy addresses mulating and realizing public policies?
societal issues; our analysis supports our conclusion that RE 1.1. Do the challenges have analogs in RE for software in-
techniques could be utilized to support policy creation and im- tensive systems?
plementation. (Abstract) 2. How could RE techniques help mitigate the identified pub-
Index Terms—Requirements engineering, public policy, bias, lic policy challenges?
unintended consequences, mainstream media, ideology and belief,
2.1. Can RE techniques be used to proactively identify
failure. (key words)
possible public policy challenges during formulation
and before enactment?
I. INTRODUCTION To answer these questions, we performed an explorative
We believe that there is a strong parallel between crafting case study using North American mainstream media and cate-
public policy in response to (societal needs to meet) citizen’s gorized the motivating problem, goals and solutions for eight
goals and software crafted (in response to requirements) to topics that received significant MSM coverage over the study
meet stakeholder goals. In this context, we define public policy period. The study materials were gathered by monitoring news
as the mechanism through which societal challenges are identi- feeds (e.g. Google News) for a period of approximately one
fied and addressed by the creation of policies, laws and regula- year and capturing those documents that presented a public
tions as enacted by government. We see sustainability as a sig- policy issue along with analysis or commentary. We reviewed
nificant societal challenge that could be addressed by effective the documents en masse, then categorized and coded them.
policy creation and implementation. Our analysis revealed evidence of (apparently unintentional
Requirements Engineering (RE) practices such as goal and often large-scale) side effects. These unintended artifacts
identification and modeling, requirements analysis, require- appear to exhibit many of the classic RE problems that occur
ments negotiation, prioritization and triage have direct corre- during the development of software-intensive systems.
spondence with the political process of policy identification, The rest of this paper is organized as follows. In Section 2
policy creation and with resolving challenges associated with we review prior and related work. Section 3 presents the re-
realizing policy goals [16]. What is not as clear is the corre- search methodology, research design and discusses threats to
spondence between RE practices associated with identifying validity. Section 4 describes the data collection and analysis
risks, threats and unintended consequences [11], and develop- efforts and Section 5 presents our observations. A supplemen-
ing appropriate mitigation strategies for the proposed policies. tary discussion follows in Section 6 and Section 7 presents the
Unintended consequences and mitigation strategies are particu- conclusions and directions for future work.
larly important for sustainability initiatives.
Copyright held by the author(s).
II. PRIOR AND RELATED WORK Wright and Hinson [54] analyze the impact of social media
We present related work from the topic areas of ideological on public relations practices and conclude that traditional news
biases in stakeholders, mainstream media as information source media still receive higher credibility than social media.
in RE, challenges of data mining versus humanism, and prob- Maalej [30] and Pagano [33] have used app store reviews to
lem analysis in other domains using RE tools. extract requirements. Guzman and Maalei [19] found sentiment
analysis to be very insightful. App store reviews are signifi-
A. Ideological biases in stakeholders cantly different from the mainstream media analyzed in this
The works on ideological biases in stakeholders are princi- paper, but also use public opinions for informing RE practice.
pally in the area of business policy. For example, in 1986, Guzman and Maalei also investigated Twitter messages to
Shrivastava [45] discusses whether strategic management is understand their potential to help requirements engineers better
ideological, and reviews 20 years of strategic management and understand user needs, using the micro-blogging system as an
business policy research and practice. He points out critical additional information source for RE. In contrast, our research
criteria like the denial of contradiction and conflicts as well as uses RE analysis to understand parallels between RE for soft-
the naturalization of the status quo, and advocates for an open ware intensive systems and crafting public policy.
conversation between managerial interests and societal stake-
C. Challenges of data mining versus humanism
holders of organizations. Parts of such an open conversation,
albeit very limited, are mass media articles like the ones ana- Manovich [31] discusses the promises and challenges of big
lyzed in the current work. social data with the optimistic conclusion that the new, en-
Handelmann et al. [25] discuss ideological framing in larged surface and enlarged depth could facilitate asking new
stakeholder marketing based on a longitudinal analysis of types of research questions.
stakeholder dynamics in more than 2,000 articles from 45 years Kirschenbaum [28] explores the opportunity of using data
of grocery retail trade. They conclude that the interpenetration mining for literary criticism in digital humanities. Kirschen-
of strategic and institutional factors has implications for stake- baum rightfully argues that literary criticism rarely uses ground
holder marketing. This ideological influence on institutions is truth, and that data mining could point out outliers that ‘pro-
also detectable in the media analyzed in our study. voke’ human subject experts. The authors conclude that “While
Entine [16] critiques the myth of social investing based on there will hopefully always be a place for long, leisurely hours
an analysis of the flaws of proclaimed objective ratings and of spent reading under a tree, this is not the only kind of reading
‘socially responsible’ businesses and their strategies. He con- that is meaningful or necessary.” (p. 5) [28] This result may
cludes that social screening is highly anachronistic and based indicate that the current work may be observing some, or all, of
on ideologically constructed notions of corporate social respon- the same characteristics.
sibility. Taking a stance against Entine’s analysis, Waddock Sculley and Pasanek [42] investigate the impact of implicit
[52] explores the myths and realities of social investing and assumptions in data mining for the humanities and argue that
provides evidence of the objectiveness of the ratings while not- the standards for evidence production in digital humanities
ing their remaining issues. We see similar tendencies of cri- should be even more rigorous than in empirical sciences. Their
tique and counter-critique in some of the news articles we ana- most important conclusion is to keep the “boundary between
lyzed – two sides with reasonable arguments, and the use of computational results and subsequent interpretation as clearly
inflammatory terms elicits stronger responses from the public. delineated as possible.”
B. Mainstream Media as Information Source in RE D. Problem analysis in other domains using RE tools
Chomsky [14] discusses what makes mainstream media Chandrasekaran [12] provides a task analysis of design
“mainstream”. He elaborates that most of mass media is in- problem solving. Byrd et al. [8] synthesize research on re-
tended to divert attention (consumers as spectators), the elite quirements analysis and knowledge acquisition techniques for
media is geared towards the educated, wealthy and powerful, management information systems.
and most academic articles are still within the boundaries of The requirements engineering community has made signifi-
institutional obedience. He concludes that, from these charac- cant contributions in the area of legislative work, for traceabil-
teristics, we can predict what we would expect to find in the ity and analysis [2][6], for resolving cross-references [38], for
current work – and we did. conformance checks [2], and for technology transfer [39].
Kwak et al. [29] compare user-generated content to main- There is further work in the legislative application domains of
stream-media-generated content, specifically around sport public governance [1], taxes [46], medical device development
communication, and concludes that message valence had a [27], procurement [40][41], nuclear [50], aviation [49], auto-
strong impact on triggering biased source evaluation and atti- motive [29], and corporate intellectual policy [9]. The work at
tude. We see a similar tendency in the streams we analyzed. hand expands this body of work to new areas.
Newman [32] explores mainstream media and the distribu- Due to space restrictions, there are large areas of work
tion of news. He highlights the contribution of social media to within RE which this work has not referenced.
social discovery and their function as network nodes for social III. RESEARCH METHODOLOGY
distribution – and points out the disruptive effects this has on
the business models of news organizations. We conducted an exploratory case study over a period of
approximately one year during which we investigated public
policy topics where there was significant Main Stream Media • Climate change (e.g. resilience, carbon emissions, en-
(MSM) press coverage. The MSM was used as an information ergy, electric vehicles, pipelines)
source, rather than the academic literature, because we were • Controlled substances (e.g. state versus federal law,
focused upon public policy and the MSM is the principal in- avoiding crime, licensing, taxes)
formation source for members of the general public. • Equalization (e.g. income, taxes, resources, cost of liv-
The MSM was monitored using news feeds such as Google ing)
News (https://news.google.com), content aggregators that can During the coding phase, we attempted to identify the chal-
be trained (via click through on articles) to perform some de- lenges that the policies were meant to address and the subse-
gree of filtering upon the vast quantity of available material. quent problems that arose because of the policy implementa-
Whenever we identified an article related in some way to an- tion. We then mapped the results to traditional RE nomencla-
nounced public policy and the author’s commentary identified ture (e.g. in some articles we found indicators of inadequate
inadequate results or unintended consequences, we then cap- stakeholder identification). A sample of the coding sheet is
tured that article to the document repository for later analysis. presented in Table 1.
The resulting dataset is a collection of 152 articles or doc-
uments on government policies, policy topics or policy initia- A. Threats to Validity
tives, government procurement and policy implementation This study has several validity threats that need to be dis-
strategies. Sustainability was the primary focus of 37 of the cussed. One of the significant construct validity threats is the
articles or documents. In each case, the topic of the article was assumption that RE processes and policy crafting processes
an initiative that was (seemingly) made with the best of inten- share a strong parallel. We believe that the collected evidence
tions. Unfortunately, the results ranged from simply inadequate and discussion presented in the paper provides sufficient evi-
to outright failure and the incidence of (potentially large-scale) dence to support our claims. Still, further empirical validation
unintended consequences was high. We include in the category of this assumption needs to be performed in the future.
of unintended consequences, policies that even a superficial RE The most significant threat to internal validity is that the
analysis would identify as probably not achievable given the observed unintentional effects and consequences have not been
solution constraints. The unintended consequences were either statistically analyzed or confirmed. We have not used experi-
explicitly identified by the author of the article or they were mental methods to study the effect of changes in the independ-
identified after our own analytic efforts (e.g. diverging or con- ent variables on the dependent variables (for example, involv-
tradicting policy goals) or prior domain experience. ing a class of stakeholders in relation to unintended conse-
As a counterpoint to the MSM sources, we also investigat- quences). However, the study has an exploratory nature and we
ed sustainability policies in California, USA [9][36]. We had do not claim that the presented consequences are complete or
access to very detailed policy and implementation plans that true for all contexts.
had large investments in their development and which we ex- Conclusion validity threats have limited impact on this
pected to be of significantly higher quality than the MSM doc- study since we have not used statistical tests to obtain our re-
uments and to be relatively bias-free. sults. At the same time, We made several efforts to minimize
At the end of the document collection phase, the documents subjectivity and resolve potential conflicts when analyzing and
were reviewed in their entirety in two sessions totaling approx- categorizing qualitative evidence. We used researcher triangu-
imately 12 hours. We used researcher triangulation to decrease lation to decrease the subjectivity bias, with the first two au-
the subjectivity bias, with the first two authors performing the thors performing the analysis in discourse and the third author
analysis in discourse and the third author reviewing the coding reviewing the coding and interpretation for consistency and
and interpretation for consistency and correctness. The coding correctness.
was emergent and led to the following eight categories. Given Since the study is exploratory, external validity remains the
the topic areas, there is some potential that a document could main limitation of our work. We aim for analytical generaliza-
be coded into more than one category; the final placement was tion rather than statistical generalization [16] and present the
based on discussion among the researchers. case and method details to enable replications and further stud-
• Algorithms (e.g. big data analysis, artificial intelli- ies. Still, we studied only a limited dataset of 152 articles on
gence) that have drawn sufficient attention to warrant government policies and policy initiatives.
public policy discussions We note that we are taking a humanistic approach to our
• IT projects (e.g. large-scale publicly funded projects, analysis. While there is a significant body of research in
generally in support of some policy goal) automated processing of news feeds and sources like Twitter,
• Social (e.g. free speech, critical thinking, gender issues, that work generally analyzes large corpuses. Unfortunately,
fake news, radicalism) that is not the way “the average person works”; they do not
• Privacy (e.g. location data, social media, children’s read hundreds or thousands of articles on an issue, they might
self-determination) read one or two. This is a substantial validity threat, but we
• Policy (e.g. cybersecurity, copyright, taxes, housing) mitigated this risk by individually reading every article and
performing the final coding after discussion.
TABLE 1. DETAILED CODING SAMPLE
ID Category Goal Solution Unintended Keywords Problems
Consequences
Reuters AI Algorithms Make news faster, Use 13 AI algorithms Potential to eliminate the News, media, Stakeholders
more accurate, and that mine Twitter and jobs of 2500 highly AI, algorithm not considered
more resilient against cluster and extract educated and skilled
fake news attempts. news reporters
Amazon sales Algorithms Make customers want Sales algorithm that (1) Suggesting composites Sales, online, Unintended
algorithm to buy more and feel works well in selling that are potentially useful algorithm consequences,
well taken care of things that are useful to build explosive devices, incorrect
because frequently (2) could potentially serve interferences
bought together to detect potential terrorists
Passport Canada IT projects / Make passports safe Electronic system to At least $75 million over Passport, IT Failed to
New passport and secure with produce the new budget and well behind consider
processing physical robustness passports schedule. The project "did specific quality
system not include a plan for requirements
security requirements."
Sight of personal Privacy Software company Software can be given Loss of privacy, complete - Privacy EU has data
privacy, Google, needs to make money away for free if we unintended by the user. privacy policies
Facebook collect the users’ data Unintended consequence that would make
instead by the companies is a lot of US
regulatory pushback of startups illegal
various jurisdictions. No
understanding of
consequences (e.g. why I
didn't get the job/loan/etc.)
Controlled Controlled Get people to relax and Pot legalized. Potentially reversing Controlled A federal
substances substances not be anxious. policy. You smoke pot in substances marshal could
summary Cali, leave to EU, return, if walk across the
you deny to have smoked US and throw
upon return to the US you anyone in jail
can be charged, perhaps that has smoked
jailed. pot.
Equalization Equalization Ensure that all political A formula. It is People don't have trust that Equalization People don't
summary regions in the country criticized as being far equalization is fair. Some bother to learn –
are able to provide too complex. From a regions experience greater what it costs to
approximately mathematician's levels of taxation than deliver a public
equivalent public perspective the model others. The people who policy initiative.
services to the citizens is grossly inadequate, receive the money say Stakeholders
inaccurate, and everyone gets taxed the can’t understand
simplistic. First year same, but the federal a correct
calculus is way more government takes that solution so they
complex than this money and reallocates it as (grudgingly)
model, but the general rebates to different regions, accept a flawed
public still perceives it so it has the same effect as solution.
as too complex. different taxation.
California California California aspires to be How many things are Millions of dollars are Sustainability Prioritization
Sustainability Sustainability a thought-leader and being considered at being spent on these policy
Policies summary Policies puts into action a lot of CSULB and the Port initiatives. What could
what we have learned of LA as examples - have been achieved if that
on mitigating shows you can make money would have been
sustainability substantial differences spent somewhere else, e.g.
challenges over the but at high costs in India or another
past years. developing country where
it can help far more people
with the same resources?
of the categories Controlled Substances, Equalization, and Cal-
ifornia Sustainability Policy, as we found the results more in-
IV. REFLECTIONS UPON THE METHODOLOGY sightful on the aggregated level. For each row, we identify the
We retrieved and analyzed 152 articles and an excerpt of Goal as the original intention for the policy and the Solution
our analysis is presented in Table 1. The left column indicates that was chosen. We then identify the Unintended Consequence
the identifier of the news item, then the category into which we arising from that solution. We further tagged with Keywords
classified the article. The bottom three rows are summary rows and identified Problems of the scenario.
For example, the first row identifies the issue of the Reuters V. OBSERVATIONS
AI, in the category Algorithms, where the decision makers had We begin each grouping of our results with a descriptive
the intention of making news faster, more accurate, and more label, present our observations and, generally, present one or
resilient against fake news attempts (in response to public out- more (sometimes rhetorical) questions.
cry and nascent public policy initiatives). The established solu- Legislative Contradictions: We observed cases where
tion was to deploy 13 AI algorithms that mine Twitter feeds to there are contradictions within legislation – how do the indi-
identify topics of interest. The (potentially) unintended conse- viduals responsible create legislation that contradicts? Are
quence of the desire to more quickly react to current events is these conflicts deliberately created by those responsible or is
that the jobs of 2,500 highly educated and skilled reporters are there something else influencing (and possibly corrupting) the
potentially being eliminated. We associated the keywords process? For example, on the topic of marijuana legalization,
news, media, AI and algorithms, and the main problem that all individual states in the USA have decriminalized personal use
relevant stakeholders were not considered. while federal law continues to make possession a crime. Legis-
The retrieved articles are dominated by works wherein the lators in individual states have deliberately chosen to contradict
author reflects upon some policy initiative and the associated federal law. Would we tolerate conflicting, and unresolved,
successes and failures. These kinds of articles appear to be in- requirements when designing a software intensive system?
herently biased towards negative critique (perhaps in an effort Related work represented policy constraints as logic program
to generate more page views?) and they became a rich source, [44], but that is only a very first step in solving these issues.
perhaps even a treasure trove, of unintended consequences. Same Old Problems: Despite 50 years of experience in IT
These articles all contain strong observer bias (they are opinion systems, the last 30 years of (approximately) which RE has
pieces), but we use them as data sources anyway – for these are been a formal discipline, we observe that system after system
the same data sources that shape their reader’s opinions and continues to experience problems such as missing requirements
perceptions. After all, just because a source is biased it does not and missing stakeholders. For example, the Government of
mean that the inherent message is not reality to the reader. We Canada embarked upon the creation of a unified payroll system
also gathered resources for two instances where policy was for all federal government employees. The system must man-
reduced to practice (sustainability and environmental policy age the contracts for hundreds of different unions, each of
initiatives at California State University Long Beach and the which has their own pay scales, promotions, benefits packages
Port of Long Beach, including significant traditional engineer- and retirement plans. Individual employees could spend their
ing technology analyses and engineering economic analyses). entire career within a single union or change to a new union
The first two authors coded the articles in discussion and each time they change the position in which they are employed.
we applied significant domain knowledge of their own to pro- RE practitioners would immediately recognize the likelihood of
vide context for the observations and to enhance the richness of a combinatorial explosion in the business rules and data ele-
the conclusions. This technique has the potential to provide ments that must be managed and would identify the issue to the
greater insight but is also a significant threat to validity. We are stakeholders. In this case, the issue appears to have been trivial-
trying to be humanistic in this work, we are analytical but not ized or ignored, and while we do not have any “insider infor-
coldly so. In other words, we are reacting as people, not as a mation” that would allow us to elaborate further, we do note
machine algorithm. We are observing emotional content and that the project is considered a near-complete disaster by all
there is the potential that we have introduced some of our own stakeholders and projected implementation and remediation
emotional bias on some of the topics. For each article, we read costs are in excess of 400% of the original budget (the Gov-
the content and (typically) the first 50 to 100 reader comments ernment recently announced1 that cost estimates have exceeded
(assuming that comments are present). $1B CDN and the creation of a task force to find a replacement
Within the first few months of our study, we realized that before this system is even fully functional). Other government
our data set would have an inherent bias: it would not be unrea- IT projects (especially those related to health care) do not seem
sonable to assert that the MSM generally reports upon things in to fare much better.
a negative manner, and the associated comments are often Holistic Perspectives: What is possible and highly desired
more extreme than the studied article. from a political perspective is often not possible from an eco-
With this realization we adjusted our research effort, focus- nomic perspective and it seems that policy makers rarely take
ing more of our efforts upon those reports wherein there ap- this holistic view. For example, promoting the use of electric
peared to be unintended consequences of some public policy cars should reduce CO2 emissions and is relatively easy to jus-
initiative. We note that there were many, many reports of unin- tify if the only metric used is emissions per distance traveled.
tended consequences and not all were negative. We then re- However, electric vehicle production creates significant CO2
fined our effort to identifying the unintended consequences and emissions [17] and the consumption of significant quantities of
evaluated them using a system model. Our analysis was based rare metals. The electricity used must be generated by low
on the question “If this was a software system and we were emission sources and (somehow) delivered to the vehicles. It is
performing a post hoc evaluation using RE techniques, what well-known in electrical engineering practice that the North
observations and recommendations would we make?”
The complete codebook is available on Google Drive [15]
1
http://www.ctvnews.ca/politics/minister-fixing-phoenix-pay-system-could-
cost-1b-1.3672663
American electric distribution grid was not designed to deliver address these differences. Despite the well-established princi-
this much energy and can only hope to do so with careful de- ple that federal powers overrule provincial powers, individual
mand management (e.g. only charge your vehicle after 9 p.m. if provinces that do not agree with federal policy on a given topic
you live in a residential neighborhood) unless significant in- can, and do, attempt to override the federal policy by crafting
vestments are made in infrastructure improvements. Despite confounding or competing legislation within those aspects of
the challenges, to achieve the desired public policy objectives exclusive provincial jurisdiction. For example, inter-provincial
systemic changes and early adopters will be necessary. For this, and international pipelines are clearly placed under federal ju-
we can use the foundations of systems thinking [13][21] and risdiction. Environmental regulatory powers exist at both the
apply them to engineering activities [3]. federal and provincial levels and anti-pipeline activists attempt
Identifying the “Right” problem: Our review identified to use provincial environmental regulatory powers to impede or
numerous cases where those directly involved with a policy or completely block any federal approvals for such projects.
project appear to believe that they have correctly identified the While conflict identification and resolution have been targeted
problem, and that their proposed (or actual) solution addresses by [51][4], these policy-level conflicts require a more holistic
the problem. However, when other parties look at the problem level of modeling and reasoning than technical requirements.
they quite strongly disagree upon the problem definition – con- Privacy: Even though users legally agree to having their
sider the acrimony that exists between perspectives on climate data tracked by many of the apps and services that they use,
change challenges and proposed solutions. This pattern implies most people grossly underestimate the magnitude of the data
that there is a class of problems where perspective is very im- trail that they create. While this data collection activity is pre-
portant. If that is the case, is our established body of RE prac- sented to the user as a way to improve the user experience,
tices applicable to those problems? Does RE have to evolve to more and more users are learning that the same data can also
be able to support these problems or do we just ignore that have significant unintended consequences – especially when
class of problems? This concern is partly addressed by some that data is licensed to a third party. We routinely see reports of
work in RE on viewpoints[48], but only on a level of technical individuals that use social media experiencing negative conse-
representation in requirements documentation. Do these prob- quences (e.g. denied insurance claims, denied bank financing,
lems also affect RE for (software-intensive and other) systems? inability to get job interviews, etc.) [26].
Side Effects: When reviewing the articles, we were repeat- Public policy is responding to these reports, most notably in
edly given the impression that comprehensive analyses of po- the European Union, and there is increasing pressure on the
tential complicating factors is either performed badly or not providers of these services to support correction and deletion of
performed. This is an area where RE can significantly contrib- data collected about individual users, including the “right to be
ute beyond the work in [55]. For example, the Swedish gov- forgotten” [19] [53]. But, what about the effect that the data
ernment performed an analysis that showed that (in their opin- had on the analyses before it was modified or deleted? And
ion) too many motor vehicle accidents occurred when overtak- how does the modification or deletion request propagate to
ing (passing). To reduce the accident levels, flexible posts were third parties that may have a copy of the original data or anal-
installed in numerous stretches of the roads. While these flexi- yses that were based upon the original data? How do we con-
ble dividers may have reduced the accident rate for cars and struct requirements not just for the originating system but also
trucks, they have made travel more dangerous for motorcycle for third-parties?
riders who cannot hit these barriers without serious conse- Biases in AI and Data Mining: Bias in automated deci-
quences. sion-making systems is receiving ever-increasing public policy
(Magnitude of) Unintended Consequences: The unin- attention. Closely related to privacy issues, deliberate or unin-
tended consequences of the policies under investigation have a tentional bias has the potential for significant unintended con-
much larger scope and scale than we expected. And, larger than sequences. The specialists in the field don’t always know why
the original policy intervention necessarily would have made they are accepting the results they get out of the algorithms,
many people believe. leading to backlash from observers (“Will anyone ever write
As an example, consider affirmative action policies whose another positive story about a tech startup? I said probably not”
goals are to “level the playing field” between disparate groups. [22]). If the algorithms that are being used to mine these data
Superficially, these policies obtain at least grudging acceptance repositories have biases (intended or unintended), they may
by a majority of the populace in North American jurisdictions. amplify negative conclusions about individuals that are un-
However, we see evidence that different special interest groups founded or unwarranted. The same technologies can also be
“weaponize” these policies, in different ways, and use them to applied to induce bias in users, from addictive video game
increase conflict rather than decrease conflict. As a result, posi- properties and Facebook’s deliberate design to induce emotion-
tions become ever more polarized and compromise solutions al reward to the numerous reports of election interference in the
become more difficult to achieve. US presidential elections2 and the Brexit campaign.
Affected Domains: Different regions within a state, prov- Significant elements of the technology sector could find
ince or country tend to have different social mores and these themselves regulated, or at least required to justify or defend
can translate to differences in local legislation and increased
potential for conflicts. In Canada, legislative powers are delib-
2
erately split between the federal and provincial levels to help to https://www.theguardian.com/technology/2017/oct/30/facebook-russia-fake-
accounts-126-million
their algorithms in ways that can be comprehended by policy ple simply cannot afford to travel except in absolute necessity
makers and by the general public. What happens when the will have the effect of reducing emissions, but is this even pos-
technology sector answers “We don’t know. We just know that sible to implement in a democratic society where people need
it (seems to) work better than anything that we have done be- to travel to work? If you believe that climate change is caused
fore.”? What will happen to “technological progress” if every- by human intervention, perhaps you could target non-essential
one believes that they have “the right” to provide input any travel – for example, ban tourism. Superficially, this would
time that they believe that this class of algorithms is affecting create non-trivial reduction in emissions. However, such a poli-
their lives (or they will threaten to claim some form of oppres- cy would eliminate a significant source of income for many
sion or human rights violation)? How many companies that developing nations and seriously impede their ability to offer
rely upon data mining would find their business models at risk public services such as health care.
in such a regulated environment? This is a significant unintended consequence. The policy
Deployment of these technologies in support of public poli- would destroy the livelihoods of everyone in the tourism indus-
cy will require us to truly understand what is “going on inside.” try and of many third world nation service industries – is that
Otherwise, how do we evaluate whether we are or not exclud- what the ‘environmentalists’ want to happen? We posit that this
ing people from fair treatment in our society based upon what is unlikely.
some algorithm returns as a result. In the world of the movie To make this point even more strongly, the Government of
“Minority Report”, precognition was combined with significant Canada attended the Paris climate convention and signed the
technical support to eliminate murder. What happens when we Paris climate accords. Later that year, the Parliamentary Budget
replace precognition with data mining and AI? The computa- Office (an agency that provides independent cost analyses of
tional techniques may be mathematically accurate but how do parliamentary initiatives) issued a report that sought to bring
we know if we are correctly interpreting the results? If, for ex- the commitments into perspective for the average citizen
ample, ethnic heritage in combination with neighborhood, so- [23][34][46]. The report identified that achieving Canada’s
cioeconomic status and educational level, leads to a person commitments would require emissions reductions of a magni-
being identified as a potential future criminal, does that mean tude that was more than the equivalent of the elimination of all
the individual is indeed a criminal and should be treated as motorized transportation in Canada – no aircraft, no shipping,
one? Does that mean that the analysis has identified fundamen- no busses, no cars, no motorcycles, etc.. How can a govern-
tal flaws or failures in society’s structure? Even though these ment maintain any credibility with its citizens if they make
factors often correlate, they are not necessarily causal and, commitments that appear to be unachievable? After all, mobili-
therefore, should we be working on fixing the cause of the ty of people and goods lies at the heart of the global economy
problem and not the symptom? Finally, if the models are telling and while the government’s actions were strongly supported by
us things we don’t want to hear, then maybe the models are the environmental movement, the average person’s position has
simply identifying opportunities for improvement. shifted toward disbelief, apathy and resentment. Rather than
Social Perspectives: We identified a pattern of hardening fostering support for the initiative, they have created resistance.
of positions by factions interested in public policy topics. Ra- In contrast, sustainability initiatives in California underwent
ther than looking for compromise, it appears that the factions significant planning efforts, culminating in realistic implemen-
are treating issues as a zero-sum game: “we adopt my position, tation plans [9][36]. Even though engineering economic anal-
or else…” Can RE techniques (especially those related to con- yses showed that some of the goals were not cost effective, an
flict resolution and mediation) be used to find common ground informed decision was made to proceed in pursuit of those
between these stakeholders? What does it mean to practitioners goals – unlike the public perception of the Canadian initiative.
(and society in general) when stakeholders tell us there can be Emotional Content: Emotionally charged content is preva-
no validity in a common ground? As Brown points out, in our lent across many of the articles, as evidenced by the author’s
current society there is a “phenomenon of you are either with selection of adjectives and adverbs and by the positions taken
us or against us.” [7] This behavior pattern, if it continues to by supporters and detractors within the accompanying com-
grow, is serious cause for concern. ment sections. From the Twitter storms of President Trump and
For example, this pattern is very evident in people’s posi- his interactions with Kim Jong-un to people issuing threats on
tions about climate change. Do you believe humans contribute social media platforms toward people who oppose their posi-
to something that is called climate change? Do you believe that tion on issues of the day, how do we get past all of that negativ-
greenhouse gasses can be absorbed by the environment without ity and unwillingness to compromise to even get to the point of
significant damage or not? Do you feel we should be minimiz- being able to agree upon a goal, let alone solutions? Are these
ing our human byproducts and pollution? behavior patterns evident even when performing RE for soft-
You can interpret these questions with sufficient qualifiers ware intensive systems?
such that eventually you will get almost every climate change Time: The time needed to introduce and pass legislation in
denier or promoter to agree. For example, many climate change support of policy initiatives (e.g. reduce industrial CO2 emis-
deniers are not against mitigation policies per se, rather they sions), and to see the effects of the policies (often measured in
tend to be against specific policies because they do not believe decades), is much longer than the average time a government
that those policies are a cost-effective solution to the problem. holds power (four to six years in most democratic countries).
For example, taxing carbon emissions at such a level that peo- This reality has led to two patterns: New governments try to
reverse policies set by previous governments resulting in abort- would be more organized and better structured than they are,
ed efforts and significant waste and, in anticipation, current but we can’t help but wonder what it would be like if they were
governments try to establish policies in such a way that they as relatively error-free as well-crafted software.
cannot be easily modified. What is lost if the original policy
implementation was actually going to be effective? How does a VII. CONCLUSIONS AND FUTURE WORK
new government undo a policy that has proven deleterious? In this work, we investigated the use of the mainstream me-
dia as a data source for an analysis of public policy and related
VI. DISCUSSION governmental initiatives. Our first research question asked
Using the MSM as a data source, rather than peer-reviewed whether we could identify challenges associated with defining,
academic papers is certainly a ‘different’ research experience. formulating and realizing public policies from this data source.
The prevalence of opinion, often strong opinion, without sub- We were able to identify challenges by analyzing the content,
stantiating evidence converts the quality evaluation process to removing the commentary, then identifying the underlying
one of (1) how well is the article written (for me)? (2) does the “facts”. We were also able to identify how people perceived the
article align with my personal biases and (3) what do I perceive challenges, which often were biased by ideology. Furthermore,
to be the reputation of the author? When one includes the we found that the challenges do have analogs in RE for soft-
comments in the analysis, it is easy to be affected by the ware intensive systems, so there is potential that RE can help to
strength of the (often negative) positions held by the comment- proactively identify possible public policy challenges.
ers. This negativity bias can easily be passed on to the analysts A metaphor that we find useful is that public policy is the
and this contagion is a known psychological phenomenon algorithm for governing the operation of the machine that is
[42].3 Researchers interested in performing a humanistic inves- society. As such it is easy to answer the next research question
tigation into these materials are advised to be prepared for the in the affirmative: how could RE techniques help mitigate the
potential emotional side-effects. identified public policy challenges? We saw no evidence that
How does the MSM affect stakeholder perceptions, opin- RE techniques could not be successfully applied in this do-
ions, and the hardening of both? There seems to be an accelera- main. Finally, we asked whether RE techniques could be used
tion and hardening of positions in mainstream media – is this to proactively identify possible public policy challenges during
something that RE might have to consider or be more cogni- formulation and before enactment? The final research question
zant of moving forward? For example, do you send your (polit- is not as easy to answer. Certainly, validation and verification
ically) left-leaning team in when you have a (politically) left- techniques could be used to identify issues and mitigate risks if
leaning client? Such a proactive effort can amplify the biases the participants had sufficient domain expertise and, dare we
(prejudices) but has the potential to lower the risk of mis- say it, wisdom.
communication. We take special note of the seeming rise in We see sufficient evidence for there to be a role for RE in
ideological bias on the part of policy makers, thought leaders helping people at large to understand the technology that seems
and the general public. This trend toward the adherence to a to overwhelm them, possible consequences and side effects.
position or interpretation independent of rational analysis of the While this work is another piece of evidence of the univer-
underlying facts could have far-reaching and unexpected ef- sal nature of problem patterns and critical thinking, it has also
fects. While we have observed evidence of ideological bias in delivered significant context for future work. Each of the major
policy, we must ask whether this trend will have an effect on points in Section V could readily become a research thrust:
RE for software artifacts. For example, will practitioners need • Legislative contradictions
to be more diligent in exploring stakeholder statements of their • Same old problems
wants, exploring whether or not these wants can be evaluated • Identifying the “right” problem
as stated (in the transition from wants to needs during prioriti-
• Side effects
zation) or whether the statements must be further explored to
• Magnitude of unintended consequences
identify ideological biases? When attempting to understand the
risks and threats arising from this trend we are prompted to ask: • Holistic perspectives
How does this knowledge inform us about how they (policy • Affected domains
makers, their constituents, and politicians) perceive circum- • Biases in AI and data mining
stances and issues; what gains, risks and threats does this offer • Social perspectives
to RE practice? In this context, miscommunication challenges • Emotional content
can potentially be greater than anticipated. • Time
We were somewhat surprised, if not shocked, by the num- Upon reflection, we must also ask whether the focus of this
ber of instances of open conflict in regulations and legislation. work is even something with which RE practitioners should be
Perhaps we were naïve in assuming that the legal structures concerned. What are, and should be, the bounds of RE? Despite
the fact we are pushing the bounds of RE, are we pushing too
far into ethics and overreaching? Are we oblivious to the fact
3
Associates of the lead author actually held what could (charitably) that there are many other people already attempting to address
be called a mini-intervention with him in an effort to determine what these issues?
had caused him to become increasingly negative over the prior six
months.
Other research questions that are prompted by our experi- [6] T. D. Breaux, M. W. Vail, and A. I. Anton. "Towards regulatory
ence, but farther afield from RE include: compliance: Extracting rights and obligations to align
• Do opinion article writers become thought leaders? Are requirements with regulations." Requirements Engineering, 14th
IEEE International Conference. IEEE, 2006.
they good barometers of the populous and their emo-
tions? [7] B. Brown, “Braving the wilderness: The quest for true belonging
and the courage to stand alone”. 2017. New York, NY: Random
• Can we use machine learning across the “wisdom of House
the crowd” as a means to validate what the pundits and
[8] T. A. Byrd, K. L. Cossick, and R. W. Zmud. "A synthesis of
politicians are saying? research on requirements analysis and knowledge acquisition
• Can we extract the core content of each document and techniques." MIS quarterly, 1992, pp. 117-138.
perform formal semantic analysis to identify the biases [9] California State University, Long Beach. “Strategic Energy
in the presentation and to quantify the intensity of the Plan.” California State University, Long Beach. July 5, 2011
bias? We could attempt to identify the ideology of the [10] David Callele, Krzysztof Wnuk. More than requirements:
policy authors, reporters and commentators, but sense Applying requirements engineering techniques to the challenge
that this is far out of our traditional field. of setting corporate intellectual policy, an experience report.
• Are things really “as bad” as the MSM would seem to RELAW 2011: 35-42
want to have us believe? Is it possible to know the rela- [11] V. Cerf, “Unintended Consequences” Communications of the
tive scale of the negative elements – were the reported ACM, vol. 61, March 2018.
negatives only a small proportion of the overall initia- [12] B. Chandrasekaran, "Design problem solving: A task
tives? Were the reported negatives only relatively rare analysis." AI magazine 11.4 (1990): 59
occurrences in the greater context of society? [13] Checkland, Peter. "Systems thinking." Rethinking management
This work has proven to be a rich source of research ques- information systems (1999): 45-56.
tions and opportunities that provide ample pointers towards [14] N. Chomsky, "What makes mainstream media mainstream." Z
future work. First, we want to deepen our analysis using empir- magazine 10.10 (1997): 17-23.
ical methods and expand on this exploratory study with a quan- [15] Results Codebook, https://drive.google.com/file/d/14ErlsN
titative analysis of evidence for said RE challenges. Second, we OOMj79GnXknWvOxu1UtG3ZUB1_/view?usp=sharing
want to use that data for a scenario analysis, applying RE tech- [16] W. N. Dunn, Public policy analysis. Routledge, 2015.
niques to explore whether we can use this form of analysis to [17] Dyer, Ezra. ”That Tesla Battery Emissions Study Maming the
prevent some of the unintended consequences that played out Round? It’s Bunk.” Popular mechanics, June 2017.
in those scenarios. Third, we plan to detail a research agenda of https://www.popularmechanics.com/cars/hybrid-
other opportunities outside of software and systems engineer- electric/news/a27039/tesla-battery-emissions-study-fake-news/
ing where requirements engineering techniques could have [18] J. Entine, "The myth of social investing: A critique of its
significant positive impact and improvement potential for the practice and consequences for corporate social performance
situations under analysis. Finally, we plan to perform an RE research." Organization & Environment 16.3 (2003): 352-368.
analysis of the goals of the GDPR, their feasibility, validation [19] EUREKA “The right to be forgotten”
and verification techniques, and monitor for evidence of unin- https://eureka.eu.com/gdpr/the-right-to-be-forgotten/
Last visited February 28, 2018
tended consequences.
[20] B. Flyvbjerg, "Five misunderstandings about case-study re-
REFERENCES search." Qualitative inquiry, vol. 12, 2006, pp. 219-245.
[21] Forrester, Jay W. "System dynamics, systems thinking, and soft
[1] A. Okhaide, M. Alhaj, D. Amyot, O. Bahy Badreddin, E. OR." System dynamics review 10.2‐3 (1994): 245-256.
Braun, N. Cartwright, G. Richards, G. Mussbacher, Creating [22] E. Griffith, “The other tech bubble” WIRED, December 16,
Quantitative Goal Models: Governmental Experience, 2017, https://www.wired.com/story/the-other-tech-bubble/
ER 2014: 466-473 Last visited February 28, 2018
[2] C. Arora, M. Sabetzadeh, L. C. Briand, F. Zimmer: Automated [23] G. Lorrie “GOLDSTEIN: Trudeau breaks his promise on carbon
Checking of Conformance to Requirements Templates Using taxing” Toronto SUN, December 14, 2017
Natural Language Processing. Trans. Software Eng. 41(10), http://torontosun.com/opinion/columnists/goldstein-trudeau-
2015, pp. 944-968. breaks-his-promise-on-carbon-pricing,
[3] Bahill, A. Terry, and Bruce Gissing. "Re-evaluating systems Last visited February 28, 2018
engineering concepts using systems thinking." IEEE [24] E. Guzman, W. Maalej. "How do users like this feature? A fine
Transactions on Systems, Man, and Cybernetics, Part C grained sentiment analysis of app reviews." Requirements Engi-
(Applications and Reviews) 28.4 (1998): 516-527. neering Conference (RE), 2014 IEEE 22nd International. IEEE,
[4] Boehm, Barry, and Hoh In. "Identifying quality-requirement 2014.
conflicts." IEEE software 2 (1996): 25-35 [25] J. M. Handelman, P. H. Cunningham, and M. A. Bourassa.
"Stakeholder marketing and the organizational field: The role of
[5] T. D. Breaux, D. G. Gordon. "Regulatory requirements institutional capital and ideological framing" Journal of Public
traceability and analysis using semi-formal Policy & Marketing 29.1 (2010): 27-37.
specifications." International working conference on
requirements engineering: Foundation for software quality. [26] J. Harris, “Take it from the insiders: Silicon Valley is eating
Springer, Berlin, Heidelberg, 2013 your soul” The Guardian, January 1, 2018
[27] H. Dagny Hauksdottir, B. Ritsing, J. Christian Andersen, N. procurement process of IT solutions for the Brazilian Federal
Henrik Mortensen: Establishing Reusable Requirements Derived Public Administration” RELAW 2014: 39-40
from Laws and Regulations for Medical Device [42] P. Rozin, E. B. Royzman. "Negativity bias, negativity domi-
Development. RELAW 2016: 220-228 nance, and contagion." Personality and social psychology re-
[28] M. G. Kirschenbaum, “The remaking of reading: Data mining view, vol. 5, 2001, pp. 296-320.
and the digital humanities." The National Science Foundation [43] D. Sculley, M. B. Pasanek. "Meaning and mining: the impact of
Symposium on Next Generation of Data Mining and Cyber- implicit assumptions in data mining for the
Enabled Discovery for Innovation, Baltimore, MD. 2007. humanities." Literary and Linguistic Computing 23.4 (2008):
[29] D. H. Kwak, K. Yu Kyoum Kim, and M. H. Zimmerman. "User- 409-424.
versus mainstream-media-generated content: Media source, [44] Sergot, Marek J., et al. "The British Nationality Act as a logic
message valence, and team identification and sport consumers’ program." Communications of the ACM 29.5 (1986): 370-386.
response." International Journal of Sport Communication, [45] P. Shrivastava, "Is strategic management ideological?" Journal
vol. 3, 2010, pp. 402-421. of management 12.3 (1986): 363-377.
[30] W. Maalej and N. Hadeer, "Bug report, feature request, or simp- [46] J. Snyder, “Canada still to fall short of Paris target”, Financial
ly praise? On automatically classifying app reviews. "Require- Post Canada, December 29 2017,
ments Engineering Conference (RE), 2015 IEEE 23rd Interna- http://business.financialpost.com/commodities/energy/canada-
tional. IEEE, 2015. still-on-pace-to-fall-short-of-paris-target-despite-ambitious-
[31] L. Manovich, "Trending: The promises and the challenges of big climate-policies-un-filings. Last visited February 28, 2018
social data." Debates in the digital humanities, vol. 2, 2011, pp.
[47] G. Soltana, M. Sabetzadeh, L. C. Briand, “Model-Based
460-475.
Simulation of Legal Requirements: Experience from Tax Policy
[32] N. Newman, "Mainstream media and the distribution of news in Simulation” RE2016: 303-312
the age of social discovery." Reuters Institute for the Study of
[48] Sommerville, Ian, and Pete Sawyer. "Viewpoints: principles,
Journalism, University of Oxford (2011).
problems and a practical approach to requirements engineering."
[33] D. Pagano and W. Maalej. "User feedback in the appstore: An Annals of software engineering 3.1 (1997): 101-130.
empirical study." Requirements Engineering Conference (RE),
[49] R. Tawhid, E. Braun, N. Cartwright, M. Alhaj, G.
2013 21st IEEE International. IEEE, 2013.
Mussbacher, A. Shamsaei, D. Amyot, S. B. Ahmadi, G.
[34] Pilling, David. “Clean electric cars are built on pollution in
Richards: “Towards outcome-based regulatory compliance in
Congo”, Financial Times, July 2017
aviation security”, RE 2012: 267-272
[35] Canada’s Greenhouse Gas Emissions: Developments, Prospects
and Reductions, http://www.pbo- [50] E. J. Uusitalo, M. Raatikainen, M.Ylikangas, T. Männistö,
dpb.gc.ca/web/default/files/Documents/Reports/2016/ClimateCh “Experiences from an industry-wide initiative for setting
ange/PBO_Climate_Change_EN.pdf metadata for regulatory requirements in the nuclear
[36] Ports of Long Beach and Los Angeles. “Roadmap for Moving domain”, RELAW 2014, pp. 2-9.
Forward with Zero Emission Technologies at the Ports of Long [51] Van Lamsweerde, Axel, Robert Darimont, and Emmanuel
Beach and Los Angeles.” Roadmap for Zero Emissions – Letier. "Managing conflicts in goal-driven requirements
Technical Report, August, 2011. engineering." IEEE transactions on Software engineering24.11
[37] Birgit Penzenstadler and Jörg Leuser: Complying with Law for (1998): 908-926.
RE in the Automotive Domain. RELAW 2008: 11-15 [52] S. Waddock, "Myths and realities of social
[38] N. Sannier, M. Adedjouma, M. Sabetzadeh, L. C. Briand, “An investing." Organization & environment 16.3 (2003): 369-380.
automated framework for detection and resolution of cross [53] Wikipedia “The right to be forgotten”
references in legal texts”. Requir. Eng. 22(2): 215-237 (2017) https://en.wikipedia.org/wiki/Right_to_be_forgotten,
[39] N. Sannier, M. Sabetzadeh, L. C. Briand, “From RELAW Last visited February 28, 2018
Research to Practice: Reflections on an Ongoing Technology [54] D. K. Wright, M. D. Hinson. "An updated look at the impact of
Transfer Project”. RELAW 2017: 204-208 social media on public relations practice." Public relations
[40] G. Lamartine da Silva Barboza, F. Cysneiros Filho, R. A. C. de Journal, Volume 3, 2009, pp. 68–73.
Souza, “Towards Legal Compliance in IT Procurement Planning [55] Yen, John, and W. Amos Tiao. "A systematic tradeoff analysis
in Brazil's Federal Public Administration”. RELAW 2016: 229- for conflicting imprecise requirements." Requirements
238 Engineering, 1997., Proceedings of the Third IEEE International
[41] G. Lamartine da Silva Barboza, F. Cysneiros Filho, R.A. C. de Symposium on. IEEE, 1997.
Souza “Towards a legal compliance verification approach on the .