Previous work has suggested that Fuzzy ARTMAP (FAM)-based Technology-Assisted Review (TAR) achieves viable levels of recall for eDiscovery (>75%), and produces models that are explainable via graphical and textual interpretation. However, these results also indicated room for improvement in recall performance, as FAM is sensitive to its training input. We evaluated the viability of improving recall performance through retraining the model based on documents evaluated as relevant from all prior review iterations. Retraining improved recall significantly, resulting in 72.9% of topic-vectorizer pairs being over the 95% recall threshold as compared to 42.2% without retraining. In addition, the FAM-based model continued to demonstrate self-stopping behavior even with retraining.
Fuzzy ARTMAP is a supervised classifier, one of many neural network algorithms derived from Adaptive
Resonance Theory (ART), which maps inputs to category labels [
Model Model 0 Model 1 Model 2 Model 3
Initial Training Source
Randomly chosen 10 Relevant, 90 Non-Relevant documents
Random order of all evaluated Relevant documents in iterations 0 to i
Random order of all evaluated Relevant documents in iterations 0 to j
Random order of all evaluated Relevant documents in iterations 0 to k
Review Iterations
0 to i
i+1 to j
j+1 to k
k+1 to stop
instead of the binary set union operator, to work with values on the interval [
While Fuzzy ARTMAP has significant benefits in terms of online learning and interpretability, it is
sensitive to the ordering of training examples [
In previous work, we evaluated Fuzzy ARTMAP performance in TAR and found robust recall
performance; however, there were several instances where Fuzzy ARTMAP failed to achieve 75% or better
recall [
We set the number of retrainings at three, established through a mix of cost balancing and small empirical tests. In general, the baseline, with no retraining, had runs that took a range of times from sub-seconds to 13-minutes; with retraining, the range of times increased exponentially, more than tripling for most runs and some taking 4 to 9 hours.
Ultimately, this resulted in up to four models produced, the first (Model 0 in Table 1) was trained with
ten relevant documents and ninety non-relevant documents. This model was used and updated with
the relevance judgements in each iteration until no more relevant documents were predicted in the ith
review iteration. A new model was then created from all the documents evaluated as relevant in the
prior review iterations, presented in a random order to the model (e.g. Model 1 in Table 1). The process
was repeated up to the limit of three additional models, and ended when the final model predicted no
more relevant documents. The rest of the model implementation was kept the same as the baseline in
[
We maintained the same hyperparameter values from our previous work, a baseline vigilance ( ) of
0.95 and a fast learning rate ( ) of 1.0. For these early results, we evaluated the retraining modifications
with the 20 Newsgroups and Reuters-21578 corpora, vectorized with tf-idf, GloVe in 300-dimensions, and
Word2Vec in 300-dimensions. All topics were used from 20 Newgroups, and the 119 topics with relevant
documents were used from the Reuters-21578 corpus, yielding 417 samples overall. Word vectors
for each document were averaged to produce document representations [
The efect of retraining on recall performance was substantial. The average diference in recall, and other metrics, between retraining and no retraining is shown in Table 2. Statistical significance was calculated using a one-way paired t-test, with p < .001 across all corpora and vectorizers for recall indicating statistically significant diference with retraining. The smallest improvement was six percentage points of recall, with most improvements in the 15 to 19 percentage point range, up to a maximum of 39 percentage points improvement in recall. This improvement in recall is further illustrated in Table 3, where 72.9% of topic-vectorizer pairs (304 of 417) achieved 95% recall or better with retraining as compared with 42.2% without retraining. Commensurate with this improvement in recall performance is a decrease in precision, mostly between 13 and 21 percentage points. This was a statistically significant decrease in precision across corpora and vectorizers (p < .001), as calculated by a one-way paired t-test.
In general, while F1 performance did decrease by a practical and statistically significant amount (p < .001, except for 20 Newsgroups-GloVe), the drop was generally more modest, with the improvement in recall ofsetting the drop in precision . Also interesting was the general increase in the average diference in the number of documents required for review to reach 75% recall; illustrated in Table 2 under the RE-75 metric. This is partly due to the number of un-retrained topic-vectorizer pairs that never reached 75% recall, compared with the retrained instances (24.46% vs. 3.36%). The increase is relatively modest for the Reuters-21578 corpus, around 300 documents or about 1.5% of the 19,044 documents with bodies. For GloVe and Word2Vec the diference is more substantial in 20 Newsgroups, around 1,400 documents or 8.5% of the 16,330 distinct posts. The efect of the retraining and number of documents reviewed is illustrated in the diference between Figure 1 and Figure 2; where more documents are reviewed, but higher recall is ultimately achieved in Figure 2 than Figure 1.
Utilizing full retraining of the Fuzzy ARTMAP model, even a modest number of times, improved recall significantly - attaining 95% recall in over 72.9% topic-vectorizer pairs with retraining as compared to 42.2% without retraining. Currently, the improvement in recall comes at the cost of precision (Table 2 - Precision and RE-75 ); however, some of the increase in the number of documents to reach 75% recall is due to the no retraining model only reaching 75% recall 75% of the time compared with 96% of the time with retraining (Table 3). This improvement in recall is achieved while retaining the characteristic that the algorithm eventually predicts no more relevant documents. Additional research is required to further characterize this stopping behavior, evaluating it across more data sets, attempting to define the statistical and theoretical basis for the predictions of no more relevant documents, and tuning the number of retraining iterations to potentially target recall levels. There is additional research opportunity in improving precision along with the improvement in recall.