Data science holds incredible promise for improving peoples lives, accelerating scienti c discovery and innovation, and bringing about positive societal change. Yet, if not used responsibly | in accordance with legal and ethical norms | the same technology can reinforce economic and political inequities, destabilize global markets, and rea rm systemic bias. In this paper I discuss an ongoing regulatory e ort in New York City, where the goal is to develop a methodology for enabling responsible use of algorithms and data in city agencies. I then highlight some ongoing work that makes part of the Data, Responsibly project, aiming to operationalize fairness, diversity, accountability, transparency, and data protection at all stages of the data science lifecycle. Additional information about the project, including technical papers, teaching materials, and open-source tools, is available at dataresponsibly.github.io.
Data science holds incredible promise for improving peoples lives, accelerating
scienti c discovery and innovation, and bringing about positive societal change.
Yet, if not used responsibly | in accordance with legal and ethical norms |
the same technology can reinforce economic and political inequities, destabilize
global markets, and rea rm systemic bias [
The public sector is under particular pressure to ful ll the mandate for
responsibility: All decisions made by algorithms will be scrutinized by the a ected
individuals and groups, and by the taxpayers who are entitled to verify equitable
resource distribution. Yet, recent reports on data-driven decision making,
specifically in the public sector, underscore that fairness and equitable treatment of
individuals and groups is di cult to achieve [
J. StoyanTovihche data science lifecycle
sharing annotation querying ranking analysis validation acquisition
curation
How can the technical community1support responsible data science practices
in complex administrative processes? Researchers are actively working on
methods for enabling fairness, accountability and transparency (FAT) of speci c
algorithms and their outputs [
To realize the limitations of this assumption, observe that additional
information and intervention methods are available if we consider the upstream process
that generated the input data [
In the remainder of this paper, I will further motivate technical work on responsible data science in the context of an ongoing regulatory e ort (Section 2). I will then highlight some work that makes part of the Data, Responsibly project (Section 3). For additional information about the project, including technical papers, teaching materials, and open-source tools, see dataresponsibly.github. io. 2
New York City is the rst municipality in the United States to attempt to
regulate the use of data-driven algorithmic decision making in government. The City
passed Local Law 49 of 2018 [
Local Law 49 of 2018 in e ect mandates the development of an algorithmic transparency framework. In the remainder of this section, I argue that meaningful transparency of algorithmic processes cannot be achieved without transparency of data.
What is data transparency? In applications involving predictive analytics, data is used to customize generic algorithms for speci c situations | we say algorithms are trained using data. The same algorithm may exhibit radically di erent behavior | make di erent predictions; make a di erent number of mistakes, and even di erent kinds of mistakes { when trained on two di erent datasets. In other words, without access to the training data, it is impossible to know how an algorithm would actually behave.
Algorithms and corresponding training data are used, for example, in predictive policing applications to target areas or people that are deemed to be high-risk. But as has been shown extensively, when the data used to train these algorithms re ects the systemic historical bias towards poor and predominately African American neighborhoods, the predictions will simply reinforce the status quo rather than provide any new insight into crime patterns. The transparency of the algorithm is neither necessary nor su cient to understand and counteract these particular errors. Rather, the conditions under which the data was collected must be retained and made available to make the decision-making process transparent.
Even those decision-making applications that do not explicitly attempt to
predict future behavior based on past behavior are still heavily in uenced by
the properties of the underlying data. For example, the VI-SPDAT [
What is data transparency, and how can we achieve it? One immediate interpretation of this term is \making the training and validation datasets publicly available." However, while data should be made open whenever possible, much of it is sensitive and cannot be shared directly. That is, data transparency is in tension with the privacy of individuals who are included in the dataset. In light of this, an alternative interpretation of data transparency is as follows: { In addition to releasing training and validation datasets whenever possible, agencies shall make publicly available summaries of relevant statistical properties of the datasets that can aid in interpreting the decisions made using the data, while applying state-of-the-art methods to preserve the privacy of individuals. { When appropriate, privacy-preserving synthetic datasets can be released in lieu of real datasets to expose certain features of the data, if real datasets are sensitive and cannot be released to the public.
An important aspect of data transparency is interpretability | surfacing the statistical properties of a dataset, the methodology that was used to produce it, and, ultimately, substantiating its \ tness for use" in the context of a speci c automated decision system or task. This consideration of a speci c use is particularly important because datasets are increasingly used outside the original context for which they were intended. This compels us to augment our interpretation of data transparency in the public sector to include: { Agencies shall make publicly available information about the data collection and pre-processing methodology, in terms of assumptions, inclusion criteria, known sources of bias, and data quality.
Data transparency is important both when an automated decision system is interrogated for systematic bias and discrimination, and when it is asked to explain an algorithmic decision that a ects an individual. For example, suppose that a system scores and ranks individuals for access to a service. If an individual enters her data and receives the result | say, a score of 42 | this number alone provides no information about why she was scored in this way, how she compares to others, and what she can do to potentially improve her outcome.
To facilitate transparency, the explanation given to an individual should be interpretable, insightful and actionable. As part of the result, data that pertains to other individuals, or a summary of such data, may need to be released, for example, to explain which other individuals, or groups of individuals, receive a higher score, or a more favorable outcome. This functionality requires data transparency mechanisms discussed above. 3
The goal of the Data, Responsibly project is to develop a foundational
understanding of responsible data science at all stages of the data lifecycle [
Algorithmic decisions often result in scoring and ranking individuals to
determine credit worthiness, quali cations for college admissions and employment,
and compatibility as dating partners. While automatic and seemingly objective,
ranking algorithms can discriminate against individuals and protected groups,
and exhibit low diversity. Furthermore, ranked results are often unstable | small
changes in the input data or in the ranking methodology may lead to drastic
changes in the output, making the result uninformative and easy to
manipulate. Similar concerns apply in cases where items other than individuals are
ranked, including colleges, academic departments, or products. Finally, even in
cases where both the data and the ranking method are publicly available, ranked
results may still be di cult to interpret. [
In addition to being commonly used in the analysis and validation stage of the data science lifecycle, set selection and ranking are also very common upstream from data analysis, in data sharing, acquisition, integration, and querying (see Figure 1), making this family of methods particularly important to study. 3.1
In [
In this work we focused on datasets in which items have a single binary sensitive attribute, such as male or female gender, and minority or majority ethnic group, with one of the groups designated as the protected group (the groups that experienced a historical disadvantage). We proposed a family of fairness measures, quantifying the relative representation of protected group members at discrete points in the ranking (e.g., top-10, top-20, etc.), and compounding these proportions with a logarithmic discount, in the style of information retrieval.
Score-based set selection is a mechanism that closely related to ranking.
Selection algorithms usually score individual items in isolation, and then select
the top scoring items. However, often there is an additional diversity
objective | selecting high-quality items that have di erent attributes (as in product
recommendation systems), or highs-scoring individuals who belong to di erent
demographic, geographic or socio-economic groups (as in college admissions and
hiring). In a recent work [
Our experimental evaluation lead to several important insights in online set selection. Most importantly, we showed that if a di erence in scores is expected between groups (e.g., due to historical disadvantage), then these groups must be treated separately during processing. Otherwise, a solution may be derived that meets diversity constraints, but that selects lower-scoring members of disadvantaged groups. This insight supports the argument of responsibility by design.
In a recent follow-up work [
We studied this phenomenon using datasets that comprise multiple sensitive attributes. We then introduce additional constraints, aimed at balancing ingroup fairness across groups, and formalized the induced optimization problems as integer linear programs. Using these programs, we conducted an experimental evaluation with real datasets, and quanti ed the feasible trade-o s between balance and overall performance in the presence of diversity constraints.
Finally, we considered the design of fair score-based ranking functions in [
We considered ranking functions that compute the score of each item as a weighted sum of (numeric) attribute values, and then sort items on their score. Each ranking function can be expressed as a point in a multi-dimensional space. For a broad range of fairness criteria, including proportionality, we showed how to e ciently identify regions in this space that satisfy these criteria. Using this identi cation method, our system is able to tell users whether their proposed ranking function satis es the desired fairness criteria and, if it does not, to suggest the smallest modi cation that does. Our extensive experiments on real datasets demonstrated that our methods are able to nd solutions that satisfy fairness criteria e ectively (usually with only small changes to proposed weight vectors) and e ciently (in interactive time, after some initial pre-processing). 3.2
Decision making is challenging when there is more than one criterion to consider. In such cases, it is common to assign a goodness score to each item as a weighted sum of its attribute values and rank them accordingly. Clearly, the ranking depends on the weights used for this summation. Ideally, one would want the ranked order not to change if the weights are changed slightly. We call this property stability of the ranking. A consumer of a ranked list may trust the ranking more if it has high stability. A producer of a ranked list prefers to choose weights that result in a stable ranking, both to earn the trust of potential consumers and because a stable ranking is intrinsically likely to be more meaningful.
In a recent paper [
In a recent paper we presented Ranking Facts, a Web-based application that
generates a \nutritional label" for rankings. [
The Recipe widget succinctly describes the ranking algorithm. For example,
for a linear scoring formula, each attribute would be listed together with its
weight. The Ingredients widget lists attributes most material to the ranked
outcome, in order of importance. For example, for a linear model, this list could
present the attributes with the highest learned weights. Put another way, the
explicit intentions of the designer of the scoring function about which attributes
matter, and to what extent, are stated in the Recipe, while Ingredients may show
additional attributes associated with high rank. Such associations can be derived
with linear models or with other methods, such as rank-aware similarity in our
prior work [
The Stability widget explains whether the ranking methodology is robust on the given dataset. An unstable ranking is one where slight changes to the data (e.g., due to uncertainty and noise), or to the methodology (e.g., by slightly adjusting the weights in a score-based ranker) could lead to a signi cant change in the output.
The Fairness widget quanti es whether the ranked output exhibits
statistical parity (one interpretation of fairness) with respect to one or more sensitive
attributes, such as gender or race. The Diversity widget shows diversity with
respect to a set of demographic categories of individuals, or a set of
categorical attributes of other kinds of items [
This work was supported in part by NSF Grants No. 1926250 and 1916647.