As a way to tackle Task 1A in CL-SciSumm 2016, we introduce a composite model consisting of TFIDF and Neural Network (NN), the latter being a adaptation of the embedding model originally proposed for the Q/A domain [2, 7]. We discuss an experiment using a development data, results thereof, and some remaining issues.
This paper provides an overview of our efforts to tackle Task 1A at CL-SciSumm 2016, whose stated goal is to locate part of a reference paper (RP) most relevant to a given citation made by a citing paper (CP). To give an idea of what it is about, consider Figure 1.
In it, you have a sentence that reads:
On the other hand, researchers from the visualization community have
designed to a number of topic visualization techniques [
Your job is to find passages in the relevant literature (what the authors call 9, 16, 17, and 18), which are most pertinent to the sentence in question. (We denote a passage in referred-to papers linked with a citation by a citation target or simply target, below and throughout the paper.)
As a way to solve the task, we work with a hybrid of two models: one that is based on TFIDF and another on a single layer Neural Network (NN). Formally, the present approach looks like the following.
(d; r) =
h(d; r) + (1
)t(d; r)
(1)
where h represents a neural network and t a TFIDF based model; d is a citation
instance and r a sentence in RP.3 For a given citation instance d, we rank every sentence
r in RP in accordance with (while dismissing those with two or less words) and select
two highest ranked sentences as a target for d. We then remove redundancies in the
output with an MMR-like measure: we take a candidate sentence off the output if its
similarity with those preceding it exceeds a certain threshold ( ). Thus, the number of
the output sentences will be further cut down to one in case they are found to contain
redundancies. In the final run, we set to 0.24 and to 0.1. We call the current setup
as ‘a neurally enhanced approach to linking citation and reference,’ or NEAL for short.
Our adding the TFIDF component to NN in is meant to compensate for the latter’s
inability to handle exact word matches effectively due to the low dimensionality of
hidden layers into which word features are mapped [
One significant consequence of using NN is that it will relieve us from the drudgery of contriving every feature that one needs to train a classifier on: NN learns by itself whatever feature it finds necessary to satisfy an objective function.
In what follows, we discuss the NN portion of , which is basically an adaptation
of the neural embedding models [
The job of NN is to provides a scoring function h that favors a true target over a false
one: that is, to build a function that ensures that h(d; r+) > h(d; r ), where r+ denotes
a true target (a sentence humans judged as a target ) and r a false target (i.e., a sentence
not selected as a target). We define h by:
h(d; r) = G(d)>F(r);
(2)
where G(d) denotes a vector derived from d and F(r) a vector from r, through word
embedding. In order for d’s similarity with its true target (r+) to be always higher than
that with a false target (r ) [
8i;j G(di)>F(rj+) > 0:1 + G(di)>F(rj ); 4 which is tantamount to: minimize:[0:1
G(di)>F(rj+) + G(di)>F(rj )]+;
d and r, respectively.
4 ‘0.1’ represents a margin we have taken from [
We project (d) and (r) into two hidden layers, V and W, through word embedding. V is an Nv K matrix (2 RNv K ) and W an Ne K matrix (2 RNe K ), with Nv, Ne, and K indicating lengths of (d), (r) and a hidden layer, respectively. (In the test run, we set K = 30 and Nv = Ne = 15; 457.) Now we let and
G(d) = V> (d);
F(r) = W> (r):
To determine values in V and W, we launch an iterative training process. Suppose that we have the training data consisting of triples,
D = f(di; ri+; Si )gi=1:::m with di representing a citing instance, ri+ a true target and Si = fri1; : : : ; ring indicating a set of false targets for di. For each (di; ri+; Si ) 2 D, we do the following. 1. For each ri 2 Si , perform a stochastic gradient descent (SGD) to minimize: [0:1
G(di)>F(ri+) + G(di)>F(ri )]+ 2. Ensure that columns of W and V are all normalized.
We developed training data from the ‘Development-Set-Apr8’ dataset (henceforth, DSA)
[
Prior to the actual run, we conducted an experiment using DSA to see how well NEAL works. The dataset comes with ten topic clusters, each of which consists of one reference paper and a number of papers that contain citations to that paper. Following a leave-one-out cross validation scheme, we split DSA into two blocks, one containing nine topic (RP) clusters and the other one. We used the former for training NEAL (or h, to be precise), and tested it out on the remaining block. The performance was measured by ROUGE-LCS, which produces the normalized length of a longest, possibly discontiguous, string of words shared by predicted and true targets. Figure 3 illustrates a citation and a corresponding target (made available by CL-SciSumm 2016 as part of gold standard data). The area shaded in green represents a citation in CP and one in yellow a target in RP. A citation and a target can span an arbitrary number of sentences.
Some statistics on DSA are shown in Table 1. RP refers to a reference paper, #CP the count of relevant citing papers, jDj the the number of instances used for training. jEj indicates how many instances are processed over the entire span of epochs, and jT j the number of citation-target pairs we used to test NEAL. Term and document frequencies (to be used for t( ; ) in ) were collected from DSA. K, Nv and Ne parameters that define the shape of NN were set to 30, 15; 457, and 15; 457 (we also used the settings for the final run). 0.0 0.4 0.8 The test proceeded as follows. For a given pair c and t of citation and target, we rank each sentence r in RP in accordance to (c; r) and select top one or two candidates as a possible target (call it g). We then determine ROUGE-LCS for g and its true target t, average scores over a entire set of citation-target pairs that belong to a particular topic cluster. Our computation of ROUGE-LCS, however, did not include tokens with less than 5 characters and those with more than 9 characters, as they were often found to be garbled and unintelligible. We also chose not to use stemming or filter out stop words.
Figure 4 shows by cluster performance of NEAL. The horizontal axis denotes the value of and the vertical axis ROUGE-LCS scores. That affects the overall performance is clearly seen. Note that NEAL reduces to TFIDF at = 0, and turns into a full-fledged NN at = 1. Thus if NEAL’s performance peaks at > 0, it will mean that NN-enabled NEAL performs better than TFIDF, or else is just as good as the latter. We observe in Figure 4 a general tendency for the performance to climb highest somewhere between 0 and 1, suggesting the superiority of NN over TFIDF, although there are notable exceptions at E09-2208 and P06-2124 where the score peaks at = 0. We have presented what we call a ‘neurally enhanced approach to linking citation and reference’ or NEAL, describing in some detail what machinery is involved and what we found in an experiment with the development data. The results appear to suggest a moderate impact of the neural network (NN) on the overall performance. But NEAL’s performance against TFIDF is far from impressive. We suspect that its somewhat lackluster performance may have been caused by our inability to clearly demarcate true and false targets: there are some words that appear both in true and false targets, which could easily derail the classifier.
Moreover, one could argue that the results of our experiment with DSA
substantiated a concern that [
h(d; r) = G(d)>F0(r; d) F0 is an F conditioned on d, where you turn off all the words in (r) that are not found in (d). What makes the idea interesting is that it points to a possibility of embedding t into h by slightly modifying the way we build (r) and (d). While it is not clear at the moment how it plays out, we believe that the idea is definitely worth a try, and something we like to explore in the future work.