Data generated across time may not be easily comparable in its original form thus potentially leading to results that may be perceived as unfair to some. We investigate quality assessment of scholarly researchers from their curricula vitae (CVs) for processes such as hiring, promotion, and grant funding. In previous work, we demonstrated that case-based reasoning (CBR) offers advantages as a transparent methodology to assess researcher quality. Its benefits include consistency, transparency, ability to adapt to specific purposes, and ability to provide explanation. The problem we now face is how to preprocess the data from the CVs to compare researchers whose scholarly production is achieved under different conditions, different points in time, and span different career trajectory lengths. We propose strategies to deal with these aspects of time during preprocessing of the data for case representation. We use 1,000 CVs from the Brazilian Lattes database to illustrate.
There is a growing interest in relying on high quality profiling systems to conduct data
studies to, as stated by Lane [
Unfortunately, private organizations have started to explore this niche and are now
steering our future by offering research metrics that rely on incomplete and flawed
automatically crawled data [
At the 2016 International Conference on Science and Technology Indicators, these
authors proposed a CBR approach to manipulate profiling data for researcher quality
assessment [
In the proposed methodology, CBR is used to classify candidate researchers as either fit or unfit for a purpose. Purposes are characterized by features that reflect specific jobs or promotions. Each entails a series of references of quality such as publications in a journal or conference that are considered more relevant than others. The characterization of the purpose comes from the users who adopt the methodology to classify CVs of applicants. The use of CBR in this task assumes that assessing quality ultimately implies predicting future success.
In this paper, we describe the CBR implementation, and discuss three preprocessing steps that are required due to temporal aspects of the data. The first is a standard normalization step so that absolute volumes of scholarly production are replaced by relative values of productivity. This avoids the comparison of absolute numbers of production accomplished in years when conditions are different. The second aspect is recency. We analyze researchers’ accomplishments to assess whether more recent production is or not more predictive of quality. The third refers to grouping the relative values of productivity depending on the lengths of career trajectories and recency. The CBR system, as it is implemented now, uses one aggregated data point for each attribute. Deciding how to group this data depends on directives of the users in terms of how they favor experience, productivity, or whether they want both to have the same emphasis.
This paper’s intended contributions are to introduce the challenges stemming from
using temporal data from CVs to assess researcher quality with CBR, and propose
preliminary strategies to address them. We illustrate these challenges and strategies with
data from 1,000 CVs from the period 2001 to 2014 from the Brazilian Lattes database
[
In the next section, we provide the background for this work, including how we proposed to use CBR for assessing researcher quality. We also mention a few related works in time and CBR, and in time-series prediction. In Section 3, we describe the challenges and our proposed strategies. We lay out directions of future work in Section 4. 2
In this section, we introduce some of the concepts used in this paper. We start with normalization, move to time-series approaches, and then discuss some aspects of dealing with career trajectories. In the final section, we describe the CBR approach that motivates this work.
Normalization is a method that may be used before a classification process, required
to equalize ranges of the features from different scales, in order to obtain the same
proportion between them, making features comparable [
GenericPred [
Time-series using bibliometrics data have been used to discover distinguished
researchers [
Time-series data of renal transplantation patients has been used in case-based binary
classification [
Dynamic time warping (DTW) [
Career Trajectories
The terms career and trajectories are viewed as synonyms that describe the path from
entering into the job market and its following steps [
The consideration of time when studying career trajectories is important for the
reliability of indicators and rankings. Previous indicators or metrics that attempted to
define a fixed interval of years have been highly criticized [
Our main problem is that this process must be transparent and able to substantiate its fairness. The assessment has to clearly consider and describe separately the biases that come from the description of purpose from the biases that originate in learning methods. The first issue we investigate is how to demonstrate whether an assessment can be fair when quality assessment is case-based, which requires comparison between researchers whose career trajectories span different intervals. 2.2
Purpose-Oriented Case-Based Researcher Quality Assessment
The purpose-oriented CBR approach classifies researchers as fit or unfit for the
purpose of a target process (Fig. 1) such as hiring or promotion [
A purpose-oriented approach requires users to input the purpose as a set of standards or examples. For instance, for a target process to hire a researcher for the federal uni-versity of Rio de Janeiro to work with the Zika virus, publications in local conferences where geographic issues are the focus may be considered of high importance when as-sessing quality of an applicant. Users can also indicate examples of fit and unfit re-searchers, which can then be used for weight learning.
The first parameter to be captured for a purpose p is the target interval of interest N, where n ∈ N is the year in question within the interval of years N that are to be included in the data from candidates to be considered for a given purpose. Years y of importance are y1 = Initial year, and yn = Final year.
Input
Classifier
Output: Classified CV Unclassified Researcher
CV data Classified researcher
CV
Classified researcher
CV Fit for a purpose
Unfit for a purpose ∑ . , 1 (1) We define weights , … , , 0,1 , for each purpose p. The local similarity measure between attribute aj, and aj’, used in the data in this article is defined by: , 1 , |a a | (2) 0, is the maximum distance between and ’ .
This way the case-based classification of fit or unfit is not assessing similarity between time-series but between flat cases with weights in each attribute stemming from the characteristics of the purpose p. 3
We use a simple example to illustrate the challenges in preprocessing data to populate cases for case-based researcher quality assessment from CV data. Suppose we plan to use our case-based quality assessment approach to classify applicants as either fit or unfit for a given purpose. One of the parameters for a job is the target interval of interest N, which delimits the years that are considered relevant to include in the examination of candidates. For example, a job opening seeking social media experts would probably not include accomplishments from candidates that predate the existence of social media. For data in Table 1, the target interval of interest is five years. Each line in Table 1 refers to the volume of one type of accomplishment (e.g., journal articles in one field and reputation) produced by job applicants (i.e., researchers).
Suppose the data in Table 1 on the left columns designated by aijn reflect all the items produced by each applicant. The maximum number of accomplishments varies each year. Only in Year 1 and Year 5, a maximum of five accomplishments was produced. In Year 3 however the maximum produced was three. We contend that there are external factors that may have contributed to higher and lower levels of productivity. One common example is a reduction of participation in conferences in periods of economic depression. We therefore normalize these absolute values and convert them into productivity rates, using Equation 3.
The results of the normalization are laid out in right columns under āijn. They show how one same absolute value (e.g., when i = 3) can represent the maximum productivity in Year 3 and 60% in Year 1. For example, if using absolute values, production of applicant in third row would be considered inferior to the applicant in the second row in Year 2 whereas relative values make them equal. This simple step is easy to describe to a broad audience and does not depend on characterization of the purpose. 3.2
Recency The challenge with respect to recency stems from the notion that recent data may be perceived as more current and therefore more relevant in time series classification. This possible perception may lead to claims of injustice and therefore we need to establish a way to assess whether or not recent data is more influential. Note that in our proposed approach, the target process may dictate the importance of recent accomplishments. Assessing how influential recent data is would be required for implementations when the target process is neutral about recency.
Given our assumption that assessing quality implies predicting future success, it is consistent to interpret that data is influential or relevant when it is predictive. To do this, we take the target interval of interest N and set the last year aside as actual to provide outcome classes. The intuition is that if a given year’s data has cases that correctly predict the actual year then this year’s data are predictive and hence influential.
To demonstrate this proposed analysis, we start from a hypothetical purpose,
namely, a job opening that seeks a researcher who is a successful collaborator. This
hypothetical purpose was captured using rules that assigned more importance to
publications and funded projects achieved in collaboration than to solo authored
accomplishments. The data where we applied these rules to determine who was fit or unfit for a
collaborative job was selected from the Brazilian Lattes database [
For the analysis we now describe, we use weights learned in [
For the target interval of interest from 2001 to 2014, we set aside 2014 as actual to
provide outcome classes. Our goal is to assess how predictive the data from years 2001
to 2013 are. For each year, we use our case-based implementation with leave-one-out
cross validation (LOOCV) [
Table 2 shows the average accuracy (AA) for all 1,000 researchers using their data from each year in the first row. The second and third rows present respectively accuracy of fit (i.e., ratio of true positives) and accuracy of unfit (i.e., ratio of true negatives).
These results in Table 2 are difficult to interpret because we do not know if the averages include the same or different researchers. To better understand these results, Fig. 2 plots true positives, true negatives, false positives, and false negatives. Positives are represented with continuous lines, and negatives are represented with dots. Light color is for true and dark for negative. A consistent trend would have true positives and true negatives in a direction opposite to false positives and false negatives. This would mean that accuracy increased because, for example, the true positives increased because of a reduction of false negatives. If accuracy increased in more recent years, then we would have to increase the relative relevance of these years when aggregating these values. The lines in Fig. 2 do not show consistent results. Our conclusion is thus that there is no consistent trend supporting the interpretation that recent data is more or less predictive than older data. Hence, ∀ āijn, N= {2001,..., 2013}, gn = 1. Values for gn are used in the next step when values are aggregated. 3.3
Career Trajectories
A researcher ri R has a career trajectory CT that reflects the researcher’s years of activity. For aggregating researchers’ production, we need:
CTi = y Max (CT i) = ymax Min (CT i) = ymin
This paper introduces time-related challenges faced when implementing CBR for researcher quality assessment. We propose a standard normalization to compare productivity instead of absolute volume of accomplishments, strategies to aggregate production across different career trajectories, and an analysis of predictiveness to address recency.
Given that there is no consensus on how many years should be used to assess researcher quality, we propose to use predictiveness of data within a target process context as a proxy to how influential it should be. We showed an illustrative example where data did not reveal variations in its level of predictiveness.
This work is very preliminary. The next step is to study different datasets to determine how to assess predictiveness and how to compute a measure of recency for when data reveals consistent trends.
The approach proposed in this paper aims to enhance the case-based researcher
quality assessment proposed in [
Given that normalization strategies interfere with classification accuracy, we need to experiment with various purpose scenarios and normalization strategies to assess which have both high accuracy and acceptable substantiation. Along these lines, we will investigate DTW particularly when comparing career trajectories of different lengths.
This paper does not detail how the characterization of a purpose may be captured, which can be through examples, conditions, and a combination of these. We also limit the presentation to binary classification and do not discuss how to produce a ranking of the applicants. These are both topics for future work.
Authors thank the STELA Institute, particularly Rudger Taxweiler for his help collecting data. First author is supported by Brazilian’s Goiás Research Foundation (FAPEG) and University of the State of Goiás (UEG) under agreement number 201310267000099. Authors also thank the suggestions from the reviewers.