Reward learning algorithms utilize human feedback to infer a reward function, which is then used to train an AI system. This human feedback is often a preference comparison, in which the human teacher compares several samples of AI behavior and chooses which they believe best accomplishes the objective. While reward learning typically assumes that all feedback comes from a single teacher, in practice these systems often query multiple teachers to gather suficient training data. In this paper, we investigate this disparity, and find that algorithmic evaluation of these diferent sources of feedback facilitates more accurate and eficient reward learning. We formally analyze the value of information (VOI) when reward learning from teachers with varying levels of rationality, and define and evaluate an algorithm that utilizes this VOI to actively select teachers to query for feedback. Surprisingly, we find that it is often more informative to query comparatively irrational teachers. By formalizing this problem and deriving an analytical solution, we hope to facilitate improvement in reward learning approaches to aligning AI behavior with human values.
for language language models [
Reward learning techniques enable AI systems to learn Bayesian reward learning algorithm that actively selects
their objectives by observing and interacting with hu- which teacher to query based on the noisiness of their
mans instead of requiring their designers to specify these feedback and the learner’s current belief. We find that
objectives manually [
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License choose , and this allows the learner to infer that the CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) gap between and is small. Section 3 formalizes this rationality model and inference procedure.
The rest of the paper is as follows. In Section 2, we
discuss prior work on reward learning, active learning, and
human modeling. In Section 3, we describe the
mechanics of reward learning, including the model of human
rationality and the metrics that will be used to measure
the value of information (VOI) of teacher feedback. In
Section 4, we propose a teacher selection algorithm that
selects which teacher to query for feedback at each time
step based on the modeled rationality of each teacher and
the learner’s belief distribution over the reward function. Figure 1: Our active reward learning approach.
In Sections 5 and 6, we present theoretical and empirical
results, showing that the learner’s belief will eventually
converge to the true reward function under the teacher velop eficient approximations to this method and show
selection algorithm, that querying less rational teachers how to integrate active query selection and reward
learncan often be more informative, and that our teacher selec- ing in practice. Lee et al. [
Human Modeling To infer reward functions, AI
systems must model the behavior of humans. Early work on
reward learning assumed that human behavior was
perfectly rational and that human teachers always chose the
alternative that maximized their reward [
We will follow recent work on learning from human
preferences [
AI systems to learn reward functions by observing or
interacting with humans. For example, inverse
reinforcement learning agents observe human behavior or policies,
and then infer an underlying reward function that the
behavior optimizes [
most informative teacher to query in order to gradually learn the correct reward model. In particular, we are interested in greedily selecting the teacher to query at each time step such that the reward model of the agent eficiently converges to the correct reward model.
At a high level, the teacher selection problem begins
with a set of items or trajectories to compare, along with a
Active Reward Learning Human feedback is expen- set of human teachers to evaluate those comparisons. The
sive and time-consuming to generate, so reward learn- human teachers each have a diferent level of rationality
ing algorithms must learn eficiently from limited data. that is known a priori, meaning that the probability of a
They do this in part by actively selecting the queries that given human teacher making a mistake by preferring a
are sent to human teachers in order to maximize the ex- less valuable item over a more valuable item is known in
pected VOI of human feedback. Sadigh et al. [
Query selection is the problem of choosing which items
to present to the teacher [
Since our focus is on teacher selection rather than query selection, for the purposes of our analysis we will assume that queries are sampled uniformly at random. However, existing methods for query selection can be easily combined with our teacher selection algorithm to further improve reward learning.
To formalize the problem of teacher selection, this section proceeds as follows. We (1) provide a representation of items and rewards, (2) apply a well-known model of human rationality to our problem, (3) ofer a method for updating belief distributions that uses preference comparisons from a human teacher, and (4) propose two metrics that measure the correctness of a belief distribution. where = − is the diference in the feature vectors of the two items.
Modeling Human Rationality Human teachers can
be represented as Boltzmann-rational agents following
a large body of existing work on reward learning [
exp( ()) exp( ()) + exp( ())
.
We thus model the human choice probabilistically: (|w; , ) =
1 1 + exp(− w⊤ ) (3) (4) where = +1 if the human prefers item over item and = − 1 if the human prefers item over item . This reflects the diference in value of the two items but not their absolute value. Equation 4 is a logistic model of the probability of the human preference , where deRepresenting Items and Rewards Intuitively, each termines the slope. As the diference in reward between item can be represented as a set of features. For example, the two items increases, the probability that the teacher a book could be described by the number of pages and chooses the higher reward item approaches 1. the number of positive reviews or a maneuver made by a self-driving car could be described by its position and Updating Belief Distributions The goal of reward distance from other vehicles at each time step. Hence, learning is to learn the weight vector w of the reward each item can formally be represented by a feature model. Given the preference of a teacher , the diference vector ∈ R where is the number of features that in feature vectors , and the teacher’s rationality padescribe the th item. rameter , the learner updates its belief over the weights
Given this representation of an item, the reward () of the reward model. That is, the belief over the weights for an item can be expressed as a dot product between of the reward model is updated such that the reward the feature vector and the weight vector w ∈ R for model now predicts that the item selected by the teacher the reward model that is being learned: is more valuable than it was prior to the belief update. () = w⊤. (1) Formally, we begin with the current belief distribution (w), which we treat as the prior distribution, and update it according to Bayes’ theorem in the following way:
approach can still be used by treating the feature vector of tahseafeoantuer-heovtecvteocrtor: wgiovuelnd tbhee1tahnditeevme,rtyhoethtehr eennttrryy (w|; , ) = ∫︀ ((|w| w′; ; , , ))(w(w′))w′ , (5) would be 0 while the th entry of the weight vector w would be the reward () for the th item. where (|w; , ) is given by Equation 4.
During reward learning, the human teacher is presented with two items and the probability of the human Measuring Belief Distribution Error After querychoosing one item over another item depends on the dif- ing a teacher and updating the belief over the weights ference in reward between the two items at hand. We of the reward model w, the belief distribution can be therefore express the diference in the reward between evaluated on a metric that measures the “correctness” or two items and as the equation the distance of this belief distribution to the true belief (2) distribution. Here, we consider two such metrics: the () − () = w⊤( − ) = w⊤ , The general form of an expected metric ℳ along with the expected metrics for mean squared error (MSE) and log loss (LL). [︁∫︀ (w)w ∫︀ (w) ‖w‖2 w − ⃦
⃦ ∫︀ (w)ww⃦⃦ 2]︁ − ∑︀ ∫︀ (w) log ∫︀ (w′)w′ ︁) w ︁( (w) mean squared error (MSE) and the log loss (LL). The use the notation w = (w), |w = (|w; , ), MSE measure represents how “far away” the belief dis- and w| = (w|, , ) throughout this section. tribution is from the true value while the LL measure represents the height of the belief distribution at the true
Importantly, the expected value of a given metric only depends on the known variables and along with value. In both cases, a lower score indicates a more accu- the current belief distribution w given a straightforwtrue||2w (6) the teacher with the lowest metric in expectation should
be selected as that would result in a weight vector that is (7) closest to the true weight vector in expectation. rate distribution. Using (w) as the belief distribution over the weight vector w and wtrue as the true weight vector, the MSE and LL measures are given as follows.
MSE((w), wtrue) = ∫︁
(w)||w −
LL((w), wtrue) = − log((wtrue))
Work on active learning from human preferences uses volume removal (i.e., removing as much of the integral of 27, 33]. However, this may not be an appropriate metric for teacher selection. This is because a larger Boltzmann rationality parameter results in a larger volume of the belief distribution being removed but may not necessarily lead to a more accurate belief distribution.
We propose a method for selecting and querying the teacher that produces the best immediate improvement in the expectation of a given metric, which approximates the expected VOI of the teacher feedback. The metrics evaluate how similar the posterior belief is to the ground truth reward, so lower scores indicate improvements in the learned reward model. The algorithm considers uncertainty over two variables: the ground-truth parameterization of the reward model and the item from the query that the teacher prefers. In particular, the expectation of the metric must be taken over the current belief distribution (w) and the probability (|w; , ) of the teacher preferring each item. Formally, we express the expectation of a given metric ℳ in Table 1. Note that we ward substitution of Equations 4 and 5. This enables our method to calculate the expected value of the metric for a given teacher with the rationality parameter . This will be used to find the teacher to query at each time step:
ric, Table 1 defines the expectations of the MSE and LL metrics using the function (w) = w/(1 + exp(− w⊤ )).
To select the teacher to query, we first calculate the expected metric for each teacher select the teacher that would result in the lowest expected metric score. Formally, the rationality parameter * that leads to the largest reduction in the expectation of the metric is defined as follows:
⎡ * = argmin ⎣ w∼ w
E ∈ ∼ |w
⎤ ︀[ ℳ(w| , w; , )︀] ⎦ , (8) where is a vector of the values of the teachers.
Learning a Reward Model To learn the reward model, the learner begins with an initial belief distribution w over the reward function parameterization and then updates it according to Algorithm 1. First, the algorithm generates queries of paired items and calculates * , which is the rationality parameter that leads to the largest improvement in the expectation over the correctness metric. The algorithm queries the teacher with this rationality parameter, and the teacher responds with a preference indicating which of the two items in the query they prefer. This preference is used to update the the unnormalized distribution as possible) as a metric [13, given the current belief distribution (w) and then 3 4 5 6 7 8 9 Algorithm 1: LearnRewardModel(· )
teachers’ Boltzmann rationality parameters , entropy convergence threshold an expected metric function E[ℳ], and an
1 converged ← 2 while not converged do
False , ← ← * ← ← (w) ← entropy ← − converged ←
GenerateQuery() − argmin ∈ E [ℳ( (w), w; , )] Teacher( * ).Query(, )
Normalize( (w) · (|w, , * )) ∫︀ (w) log (w)w entropy < 10 return (w) belief distribution w. The algorithm iterates until convergence, which is when the entropy of the distribution w becomes lower than a specified threshold .
Algorithm 1 queries multiple teachers belief distribution over w converge to the true value.
Theorem 1. In the limit of →
∞ random queries to Boltzmann-rational teachers with positive, finite values, the posterior distribution over w converges to the true value. bution over w after a sequence of queries is ∈ [± 1] from humans with rationality parameters is (|w; ) = ∏︀ =1 (|w; ). The posterior distri (w|; ) ∝
∏︁ (|w; ) (w). w ̸= wtrue. The Bayes factor between w and wtrue is
We will show that (w|; ) → 0 as → ∞ for all BF = (w|; ) (wtrue|; ) =
∏︀ (|w; ) (w) ∏︀ (|wtrue; ) (wtrue) where (wtrue|; ) is the posterior distribution at wtrue.
We can show that BF → 0 as → ∞ except when gences. Here, we show that this process will make the = 0, each of these terms equals 0. As with diferent values until the reward estimate conver- to zero when (|w; ) = (|wtrue; ). When → ∞, Proof. The likelihood of a sequence of human choices the sum is positive, so the sum diverges, and so the ︃( ︃( otherwise. Trivially, BF = 1 when w = wtrue. w = wtrue. This implies (w|; ) → 0 except when w = wtrue. We require (wtrue) ̸= 0 as BF is undefined
We now consider w ̸= wtrue. We can define the negative logarithm of BF, which approaches ∞ as BF → 0: − log (BF) = − log ︂( )︃ ∏︀ (|w; ) (w) ∏︀ (|wtrue; ) (wtrue) = − ∑︁ log ︃( (|w; ) )︃ (|wtrue; ) − log (w) )︂ (wtrue)
.
The first term is the sum of many terms. If this term approaches ∞ as → ∞ then BF → 0. We now examine each term in the sum and show that in expectation they are each positive. All of these terms are independent as they are only depend on the likelihood and not on the current distribution. Hence, they will not decay with additional steps, and so the sum will diverge if the individual terms are positive in expectation. The expected value for each term in the sum is
E − log [︃ = − (|w; ) )︃]︃ (|wtrue; ) ∑︁ (|wtrue; ) log ∈+1,− 1 ︃( (|w; ) )︃ (|wtrue; )
.
This is the KL divergence between (|wtrue; ) and (|w; ). This is strictly non-negative and only equal have the same sign. (|w; ) → (w⊤ ), where (· ) is the Heaviside step function. In this case, it holds that (|w; ) = (|wtrue; ) whenever the values w⊤ and wt⊤rue
(w|; ) → 0 for all w ̸= wtrue.
a more rational teacher (with a larger value) does not always lead to faster convergence to the true value, as measured by lower MSE or LL, because the magnitude of w⊤ can be learned from the teacher making mistakes.
demonstrate that if our current belief distribution (w) is a normal distribution characterized by and , a lower value is more informative for certain values of and .
Specifically, when the distribution is symmetric (
= 0) then a larger value of is better, and as the distribution gets broader (larger ) larger is also better. If the distribution is very wide then a large allows us to quickly 7. Experiments remove a lot of probability mass, while if the distribution is narrow (and asymmetric) then we learn about the value of w⊤ from the humans making mistakes, which requires the human to be less than perfectly rational. For example, if w⊤ > 0 then a perfectly rational human would always choose item over item , and we would not learn about the actual value of w⊤ .
We now show that our approach method for selecting
outperforms several baseline methods, using the
simple restaurant recommendation domain. In Figure 3, we
compare: (1) selecting the largest value to see if the
result that larger is not always better is true in practice;
6. Restaurant Recommendation (2) selecting randomly to ensure that the advantage
over selecting the largest is not just due to the
ranWe now discuss how our method for reward learning domness of the selection; and (3) always selecting = 1
using feedback from multiple teachers can be applied because this is often what is assumed to be the rationality
to a simplified restaurant recommendation domain. In parameter in other work.
this domain, the goal is to learn a reward function that In this experiment, the size of the weight vector is
can be used to recommend restaurants to a user. This re- = 3 and the domain of the weights is = [
More formally, the problem of restaurant recommen- generated randomly, where Cleanliness, Spiciness ∼
dation has a set of restaurants = { 1, 2, . . . , } (1, 10), and Vegan are uniformly drawn from {0, 1}.
that can be recommended to a user. Moreover, there While we generate our samples randomly in order to
isois a set of users = {1, 2, . . . , } who can late the the efect of teacher selection, any of the active
be queried about their restaurant preferences. Each query selection methods from previous work could be
restaurant is expressed as a set of features = used here. The teacher is selected and then queried using
{Cleanliness, Vegan, Spiciness} where Cleanliness ∈ one of the various methods and the belief distribution
[
For example, this analysis assumes that humans collapse to the true reward function under our algorithm.
are Boltzmann-rational decision-makers with constant, Our hope is that this method and analysis will improve
known values. While more nuanced than optimal mod- reward learning in domains where feedback is gathered
els, Boltzmann-rational models fail to account for system- from multiple teachers with varying levels of rationality.
atic biases in human judgement [