Our paper is dedicated to the selection of neuropsychological tests effective for experimental assessment of cognitive reserves, which are increasingly studied as the world population ages. In this, we separately consider the systems of attention, memory and intelligence and use the list of 36 candidate tests composed by domain expert relying on OntoNeuroLOG - Mental State Assessment ontology, which we extend in OWL format. The names of the tests are employed in extracting the publications from PubMed (MEDLINE) bio-medical database with the tools provided by E-utilities (EDirect). To assess the application context, represented as the subject group and the studied pathology, we perform word frequency analysis for the publications' titles and itemize the most prominent journals. Ultimately we select in total 5 tests that are most popularly used by the research community and whose application context is relevant for the cognitive reserves studies. These neuropsychological instruments include Stroop task, Fluency test, Divided attention tasks, Dual task, and Rey complex figure . We believe that both the mining method that we used and the resulting test battery can be of interest to experimental researchers in Neuropsychology.
Just as Mathematics has come to be “the language of science”, information technologies (IT)
are becoming “the toolbox of science”, as the volume of research publications is growing
exponentially. Moreover, the power of the growth is itself accelerating: from 2-3% annual increase
in the mid-XX century to 8-9% in 2010 [
Particularly, the issues related to the formation and activation of cognitive resources gain
in importance lately. This concept is used to explain individual differences in the changes of
cognitive functions that accompany brain ageing or damage, and the terms “cognitive reserves”,
“compensatory reserves”, etc. are used as close synonyms in various contexts and schools. The
increasing interest is due to the growing life expectancy of the population and longer professional
careers in the elder age, which require preservation of mental and physical health [
Our previous review of research results in mechanisms of the brain’s ageing suggested that
despite the seemingly existing consensus with respect to decreasing speed of mental operations,
worsening cognitive flexibility and short-term memory [
To assess the effectiveness of functions of attention, memory and intelligence systems, a wide diversity of experimental conditions and test batteries are utilized, ranging between different countries and various pathological conditions. For instance, the relatively unsophisticated Mini–Mental State Examination is certified in the Russian healthcare system, but it is mostly useful for screening obvious dementia. As for the CR assessment, the number of test batteries proposed in the literature is rather high, which impedes their selection in practice and makes cross-publication meta-analysis troublesome [7; 8]. So, the goal of the current research is the development of test battery for CR based on the identification of the experimental tests and subject groups that are most common in the studies of the systems’ functions with respect to the cognitive reserves. Particularly, we perform: 1) the extraction of the test-related publications from PubMed and their counting per the time periods, 2) the identification of the most common subject groups and the experimental contexts (particularly, the studied pathologies) from the word frequency analysis in the extracted publication, 3) the itemization of the most prominent publication venues (journals) for each of the tests.
The remaining of the paper is organized as follows. In Section 2, we describe the use of tools for PubMed mining and the domain ontology that we employed. In Section 3, we present the software implementation and the results of the analysis. Finally, we discuss the findings and their limitations and provide the conclusions.
Our mining method is based upon the analysis of the tests’ popularity in the publications related to CR, so its main assumption is the wisdom of the research community. In the next sub-chapter we detail the method and overview some related work.
Quantitative mining of existing publications in Neuroscience nowadays often has the form of meta-analysis – the study of several aggregated datasets, usually using statistical methods. A certain disadvantage of this method is that it requires access to the publications’ full texts or the openly available datasets. Also, it has poor capabilities for automation, particularly if it seeks to cover relatively dated, pre-standardization research works. Another popular kind of secondary research is bibliometrics: even though it primarily deals with impact, citations and ranking of publications, there are also applications for the development of thesauri (bibliomining) and studying the relative “impact” of concepts. In [9], the top-10 terms with the highest relative citation scores were identified based on the publications’ titles and abstracts extracted from Web of Science’s “Neuroscience” category and on the Journal Citation Reports (JCR).
Still, PubMed appears to be a more appropriate source for biomedical data and text mining, both due to the focus of the underlying MEDLINE database, the great number and temporal range of publications, and the robust API access tools. In the last decade, the emphasis on developing custom software for biomedical mining (such as [10] or [11]) has gradually diminished, as Entrez (E-utilities) become the de-facto standard [12], save possibly for Qinsight (Quertle), which however is a commercial product. E-utilities are 9 server-based programs that provide interface into the Entrez query and database system at the National Center for Biotechnology Information (NCBI). In our current research we mostly relied on the following utilities and functions2: • ESearch – responds to a text query, returning the list of matching publications identifiers (UIDs) in a given database, for later use in ESummary, EFetch or ELink. • ESummary – responds to a list of UIDs from a given database with the corresponding document summaries (DocSum). It functions for all Entrez databases, and a text search in web Entrez is equivalent to ESearch-ESummary . • ELink – responds to a list of UIDs in a given database with either a list of related UIDs and relevancy scores in the same database or a list of linked UIDs in another Entrez database. • EFetch – responds to a list of UIDs in a given database with the corresponding data records in a specified format. Currently, it does not support all the 38 Entrez databases; • efilter – navigation function that filters or restricts the results of a previous query within the Entrez databases; • xtract – Entrez direct function that converts EDirect XML output into a table of data values.
While the above utilities and functions allow extraction of publications and their specific fields from the Entrez databases, our goals also included the analysis of word frequency. The Entrez Direct (EDirect) software extends E-utilities and allows access to the interconnected NCBI’s databases from UNIX Command Line. Particularly, it already includes the function that is capable of performing straightforward counting of the words in the provided textual corpus: WordAtATime() { 2 See the documentation at https://www.ncbi.nlm.nih.gov/books/NBK179288/ sed ’s/[^a-zA-Z0-9]/ /g; s/^ *//’ | tr ’A-Z’ ’a-z’ | fmt -w 1 } alias word-at-a-time=’WordAtATime’
Another function included as a script with the EDirect software allows sorting the words’ frequencies:
SortUniqCountRank() { sort -f | uniq -i -c | perl -pe ’s/\s*(\d+)\s(.+)/$1\t$2/’ | sort -t $’\t’ -k 1,1nr -k 2f } alias sort-uniq-count-rank=’SortUniqCountRank’
As we mentioned above, our third goal involves the per-test analysis of the frequency of journals that publish the papers. Conveniently, the output for the Document Summary provided by the E-utilities in XML format contains the property FullJournalName. Since the journals’ names are standardized, we felt there was no need for a sophisticated analysis, so we relied on MS Excel’s Pivot Tables functionality to count the most frequent journals for the publications extracted for each of the tests.
So, after the mining tools fit for our purposes were selected and configured, the next challenge was to obtain the list of candidate tests for the assessment of CR, per the systems of attention, memory and intelligence.
It should be noted that the term “cognitive resources” is more frequent and had emerged earlier
than “cognitive reserves” in English-language publications. The analysis of literature containing
these keywords suggests that in the study of cognitive reserves much more attention is paid to the
brain’s structures and functions, as well as their re-organizations that activates due to traumatic
or pathological damage of the nervous system, while in the study of cognitive resources it is
paid mostly to emotional and / or motivational regulation of cognitive activity. For more detailed
review of CR in Neuroscience, the reader can kindly refer e.g. to our previous publication on the
subject [
The analysis for the selection of the tests needs to be performed independently per the systems of attention, memory and intelligence, as each of them has unique context with respect to the tests application. Since this implied the need for thesaurus (query keywords) combined with domain knowledge (the classification of the tests per the systems), we turned to the use of ontology. In biomedicine, neuroscience, neuropsychology, and etc. ontologies are seen as an effective tool for organization of concepts, theories, and models related to the brain, cognitive functions and behavior, psychological metrics and so on [13]. A remarkable number of ontologies was created and made available [14], particularly since the wide introduction of the infamous Protégé ontology editor, developed by the Stanford Center for Biomedical Informatics Research.
The results of our review of existing ontologies suggested that OntoNeuroLOG [15] apparently contains the most extensive list of instruments for subject data acquisition (i.e. tests). It is also conveniently available in OWL format and represented online at BioPortal within the Mental State Assessment ontology3. A certain disadvantage was the respectable age of the project, nearly 10 years, which suggested the need for updating, given the high dynamics in the domain of Neuropsychology. Another trouble was that the names of the tests (see in Fig. 1) were by and large not appropriate for direct usage as keywords in search queries.
The representation of the systems-related classes in OntoNeuroLOG (see in Fig. 2) was not satisfactory for our purposes: 1) the structure does not reflect domain knowledge (they concepts are just direct subclasses of the domain class) and 2) they have no relations to the tests (no links to subclasses of the Subject data acquisition instrument class).
The up-to-date list with the names of the tests relevant for CR was composed by a Neuropsychology domain expert, who also specified their relations to the systems. All of these were implemented in Protégé 4.3.0 OWL editor as the extensions of the Subject data acquisition instrument class, as shown in Fig. 3-5 (visualized with OntoGraph plug-in). The names of classes at the bottom level correspond to the keywords as they will be used in the search queries for the 3 https://bioportal.bioontology.org/ontologies/ONL-MSA/
The E-utilities (EDirect) PubMed mining tools accept natural language-based queries as the input and output matching publications, while one of the configuration options is Automated Term Mapping (ATM). Enabled by default, it extends the initial query with the supposedly relevant keywords selected from a list of pre-indexed terms. The somehow unpredictable helpfulness of ATM has long been the source for complains in the PubMed mining research [16]. However, in our informal tryouts we found that its current implementation increases the number of extracted publications by the relatively consistent 8%. Since we were mostly interested in the relative numbers between the tests, we decided that this overhead, even if its relevance is uncertain, will not significantly bias the results and kept the ATM defaults.
We installed the previously described EDirect software on a virtual server under Debian operating system and used the UNIX terminal queries in the following format to perform the publications search and filtering: esearch -db pubmed -query "cognitive AND (functions OR resources OR reserves)" | efilter -query "Attention AND Network AND Test" | efetch -format docsum | xtract -pattern DocumentSummary -element Id SortFirstAuthor Title FullJournalName PubDate > AttentionNetworkTest.txt
This example query extracts the specified fields (UID, first author’s name, publication title, journal name, publication date) for the Attention Network Test and saves the output as delimited values to the AttentionNetworkTest.txt file for further analysis. So, we ran 36 such queries – for every test’s name specified in the ontology (as illustrated in Fig. 3-5).
In analyzing the word frequency, we considered several options. Searching in the publications’ titles only is by far the fastest, but it is limited in the corpus volume. Full text search, even though the slowest, is potentially the most robust, but in turned out that full texts are not accessible for most publications, particularly the ones preceding the proliferation of the Open Access. PubMed also provides the -related key for ELink, which at some higher computational costs extends the search results so that they supposedly better represent the field, similarly to the ATM mechanism that we described previously. We informally tested the correspondence of the words distribution for one of the tests to Zipf’s law using Kolmogorov-Smirnov’s test implemented in plpva library for R (http://www.santafe.edu/aaronc/powerlaws/), and found that the hypothesis had to be rejected, unlike for the simple title search. Since some of the related results suggest that natural language texts that make sense abide by Zipf’s law in contrast to random ones [17], we ultimately decided to rely on the simple title search in our analysis of the words frequencies.
So, it resulted in the following query that uses the previously described UNIX alias commands word-at-a-time and sort-uniq-count-rank : esearch -db pubmed -query "cognitive AND (functions OR resources OR reserves)" | efilter -query " Attention AND Network AND test " | efetch -format docsum | xtract -pattern DocumentSummary -element Title | word-at-a-time | sort-uniq-count-rank > frequencyANT.txt
This example query generates a table of word occurrence counts for the Attention Network Test, sorted by frequency, and saves it to the frequencyANT.txt file.
The mining was performed in April 2020, so the results are current for that date. In Tables 2-4 we present the results of the analysis per the systems: the dynamics of various tests, the most common subject groups, the pathologies and the most prominent journals, if any. The ifgures are the absolute numbers of relevant publications found, while the subject groups and the pathologies are based on the word frequency analysis in the whole body of publications’ titles for a test. The tests are sorted by the total number of publications retrieved, and the names of the most popular tests that are the candidates for the test battery are highlighted in bold. In Table 3, the Alternate Category test is not shown, as the total number of extracted publications for it was 0. In Table 4, the Travelling Salesman, Amthauer’s Structures, Mayer-Solovey Emotional Intelligence, Guilford-Sulliven tests are not shown for the same reason, so the group of socialemotional intelligence test is not represented. Blank cells in the tables mean there were too few publications for a conclusive analysis.
The results of the analysis suggest that Fluency test and Stroop task are the most common in studies of executive attention functions. The working memory with respect to the cognitive resources is studied much more often that the other types of memory; and the most common working memory tests are Dual task and Wisconsin card sorting test. The relatively smaller number of publications dedicated to the testing of intelligence as a cognitive resource is probably due to the organizational complexity and high work effort needed for such studies, which require more time compared to the performance of separate tests of attention and memory. The most common is IQ analysis based on Wechsler Adult Intelligence Scale for patients with diagnosed schizophrenia. In Fig. 6 and Fig. 7 we show the dynamics (logarithmic scale and exponential trends are used) for the numbers of publications for the selected most popular tests related to attention and memory respectively.
The expert analysis of the publications’ titles filtered from PubMed with the concepts of the ontology that we created, has further limited the set of the publications. For instance, out of the 196 articles combining CR and Attention Network Test, the expert recognized 42 (21.4%) to be indeed relevant. About the same share (22.3%) was maintained for the publications combining CR and Color Word Interference: 25 truly relevant ones out of 112 found. So, for further facilitation of the information search and analysis, we studied the frequencies of the words encountered in the titles.
The analysis of the obtained lists has confirmed the appropriateness of the classification of tests based on the expert’s opinion. E.g., for the Attention Network Test the top positions in the list were held by attention 52, cognitive (47) and network (35). For the Fluency test, the most frequent were cognitive (480), functions (171) and executive (159). Besides, the analysis allowed identification of the most common subject groups, and the pathologies, as shown in Tables 2-4.
According to Tables 2-4, the vast majority of studies with the functions of attention, memory and intelligence being tested, are dedicated lately to researching mechanisms of Alzheimer’s decease and schizophrenia. Cognitive resources with respect to attention in children are most commonly studied with Ongoing task, while age-related memory changes are studied with Dual task and Logical Memory Test.
If the cognitive reserves are considered in the traditional way, as the neural networks of complex organization obtained during the education process and the subsequent professional cognitive activities [18], then the volume and organization of the created networks are capable of compensating their partial loss due to atrophy of the neurons and nerve fibers caused by aging and the related Alzheimer’s disease processes. According to our findings, Design Fluency Test, Divided attention tasks, and Rey complex figure are the most commonly used instruments for the neuropsychological measuring of attention and memory as the indices of cognitive reserves.
Unlike the studies focused on the search and review of socio-psychological or neurophysiological instruments for identification of cognitive reserves [19; 4] (and, accordingly the surveys and questionnaires for the education level, professional activities and lifestyle, as well as the particulars of architecture and functional activity of various brain structures measured with fMRI), our work was aimed towards classification, search and analysis of neuropsychological tools for testing the functions of attention and memory systems, whose indicators are seen as the primal psychometric predictors for cognitive reserves both in normal and pathological ageing [3; 20; 21; 18]. The multitude of approaches towards the assessment of CRs subsequently defines the diversity of the functions tested. In our work we sought to identify the psychometric methods that are most frequently used or discussed, as we did not evaluate the con-text in which the test-related terms were mentioned, and it might as well have been negative ones. Still, our informal verification of the extracted publications reveals that the works in which where the tests are utilized are quite more common than reviews or critiques.
As the results suggest, various versions of the Fluency test and Divided attention tasks and the Dual task are used more often than the others in studying of ageing-related cognitive reserves. This does not come as a surprise, since the universal processes in the ageing brain are decrease in the information transfer speed (which is reflected in the generation of ideas in the Fluency test) and the decline in effectiveness of coordination of the different neural systems when the volume of the information being processed grows (hence the diminishing performances in the Divided attention tasks and the Dual task). It seems that the relative scarcity of using the Rey complex ifgure for testing spatial memory in patients with Alzheimer’s disease in literature is due to the fact that the structure the most sensitive to age-related atrophy of nerve cells is hippocampus, which ensures the formation of new traces of memory and especially spatial memory (see review in [3; 22]).
The relatively infrequent joint mentioning of the compensatory resources/reserves & memory or compensatory resources/reserves & attention (see PubMed search results in Table 1) but the greater number of publications returned in our mining with E-utilities is seemingly due to the fact that most research works dedicated to studying the mechanisms of age-related memory weakening (10659 publications mentioning aging AND memory and 5190 for aging AND attention in PubMed during the last 5 years only) use the concept of “cognitive reserves” when discussing the discovered effects, but not as important keywords.
It remains unclear why despite the education being recognized as one of the most important indicators of cognitive reserves preventing the development of age-related dementia [23; 24], relatively little attention is given to testing intelligence. Perhaps it is due to the organizational difficulties in applying the well-known techniques (Wechsler Adult Intelligence Scale and Stanford-Binet test) in fairly large samples.
Correspondingly, the described technology for mining the literature based on the keywords of interest and the frequency analysis of concepts allows effective planning of research by selecting the most informative experimental conditions. With respect to our goals related to the search for predictors of cognitive reserves, these include such instruments as Stroop task, Fluency test, Divided attention tasks, Dual task, and Rey complex figure .
An important limitation of our study is that the list of tests’ names to be used as the keywords in the literature mining process was composed based on an expert’s subjective opinion. It is very much likely incomplete, due to the expert’s limited memory and the range of interests, which mostly covers normal, but not pathological brain ageing processes. So while the current work is focused on the most common experimental conditions for testing attention, memory and intelligence, it is possible that some more rare testing instruments may be no less informative in assessing the cognitive functions. Further, the word frequency analysis, which we used to identify the tests’ application context, was performed only for the publication titles, not the full texts. However, it is unclear whether this is indeed a disadvantage, as mining of full texts is much more complex, and may be biased by the terms not related to the paper contents, e.g. from the literature review.
Another notable limitation is that no verification of the test battery’s e ffectiveness was performed, even though our research plans include this as an important priority. Among other research prospects we see the exploration of sustainability of the effect found with each of the methods, based on the analysis of the citations (impact) of publications that mention them. We also plan to consider the efficiency of tests related to emotional regulation of behavior that is known to be an important factor in formation and usage of cognitive resources. The method based on the popularity of the publications and the mining technology that we developed lay within the mainstream computational Neuroscience [25; 26]. We believe that they, as well as the proposed test battery, can be of practical use for researchers working in experimental Neuropsychology.
The reported study was funded by RFBR according to the research project No. 19-29-01017. We also thank our Master student, Alexander Lukichev, for his assistance in EDirect programming. 8. Alenius, M. et al.: Cognitive Performance among Cognitively Healthy Adults Aged 30– 100 Years. Dementia and geriatric cognitive disorders extra, 9(1), pp.11–23 (2019). 9. Yeung, A.W.K., Goto, T.K., Leung, W.K.: The changing landscape of neuroscience research, 2006–2015: a bibliometric study. Frontiers in neuroscience, 11, 120 (2017). 10. Fontaine, J.F. et al.: MedlineRanker: flexible ranking of biomedical literature, Nucleic
Acids Res., 37, W141–W146 (2009). 11. Poulter, G.L., et al.: MScanner: a classifier for retrieving Medline citations. BMC bioinformatics, 9(1), 108 (2008). 12. Drakopoulos, G., Kanavos, A.: Tensor-based document retrieval over Neo4j with an application to PubMed mining. In 7th IEEE International Conference on Information, Intelligence, Systems & Applications (IISA), pp. 1–6 (2016). 13. Poldrack, R.A., Yarkoni, T. From brain maps to cognitive ontologies: informatics and the search for mental structure. Annual review of psychology, 67, 587–612 (2016). 14. Norris, E. et al.: A scoping review of ontologies related to human behaviour change.
Nature human behaviour, 3(2), 164–172 (2019). 15. Gibaud, B., et al.: NeuroLOG: sharing neuroimaging data using an ontology-based federated approach. In AMIA Annual Symposium Proceedings, 2011, p. 472 (2011). American Medical Informatics Association. 16. Adlassnig, K.P.: Optimization of the PubMed automatic term mapping. In Proc. Medical
Informatics in a United and Healthy Europe (MIE), 150, p. 238 (2009). IOS Press. 17. Ferrer-i-Cancho, R., Elvevåg, B.: Random texts do not exhibit the real Zipf’s law-like rank distribution. PLoS One, 5(3) (2010). 18. Stern, Y. et al.: Whitepaper: Defining and investigating cognitive reserve, brain reserve, and brain maintenance. Alzheimer’s & Dementia (2018). 19. Eich, T. et al.: Towards an ontology of cognitive processes and their neural substrates: A structural equation modeling approach. PLoS ONE 15(2): e0228167 (2020). 20. Pinto, C., Tandel, K.Y.: Cognitive reserves: Concept, determinants, and promotion. Journal of Geriatric Mental Health, 3, pp. 44–51 (2016). 21. Stern, Y.: Cognitive reserve in ageing and Alzheimer’s disease. The Lancet Neurology, 11(11), pp. 1006–1012 (2012). 22. Sharager, Y. et al.: Spatial memory and the human hippocampus Proceedings of the National Academy of Sciences USA, 104(8), pp. 2961–2966 (2007). 23. Arenaza-Urquijo, E.M., Landeau, B., Mevel K.: Relationships between years of education and gray matter volume, metabolism and functional connectivity in healthy elders.
Neuroimage, 83, pp. 450–457 (2013). 24. Boller, B. et al.: Relationships between years of education, regional grey matter volumes, and working memory-related brain activity in healthy older adults. Brain imaging and behavior, 11(2), pp. 304–317 (2017). 25. Chen, X. et al.: Topics and trends in artificial intelligence assisted human brain research.
PloS one, 15(4), p.e0231192 (2020). 26. Liu, F. et al.: Worldwide tendency and perspectives in traumatic dental injuries: A bibliometric analysis over two decades (1999–2018). Dental traumatology (2020).