People read digital documents on a daily basis to share, exchange, and understand information in electronic settings. However, current document readers create a static, isolated reading experience, which does not support users' goals of gaining more knowledge and performing additional tasks through document interaction. In this work, we present our vision for the next-gen document reader that strives to enhance user understanding and create a more connected, trustworthy information experience. We describe 18 NLP-powered features to add to existing document readers and propose a novel plug-in marketplace that allows users to further customize their reading experience, as demonstrated through 3 exploratory UI prototypes available at: github.com/catherinesyeh/nextgen-prototypes.
• A set of proposed NLP-powered plug-ins to add to existing document readers toward enhancing human-document interaction, including 12 opendomain and 6 domain-specific features. • A preliminary vision for a centralized plugin marketplace that would allow further customization of the user experience in document readers and feature development to be outsourced. • 3 exploratory UI prototypes illustrating a subset of features and the plug-in marketplace proposed for the next-gen document reader (github.com/catherinesyeh/nextgen-prototypes).
Foxit, and Sumatra PDF tend to only support static, inedxopcluomreetnhtefpeoastusirbeisli,tryeocfecnrteNatLinPgeafomrtsoraerceobnengeicntneidn,gtrtuost- aFmniegdnuuwrehco2en:ncUaeIcptptir.voeUt,osbtey)rpstehcedaencmoaro)rnetssotpgroagntliedniignngostuparllluceogdn-itpnelxuttogu-oailnltipspoopon-noulfpy worthy information experience for users. appears if relevant text is selected. (Recipe: BBC Good Food)
For example, ScholarPhi [
Sioyek [4], a document viewer designed for reading tooltip icon only appears if relevant text is selected (i.e., technical books and research papers, has some interest- text containing numerical values). Ideally, the document ing features such as smart jump for references and figures, reader would also automatically identify the correct unit searchable bookmarks, and portals to display linked infor- of measurement selected by the user and auto-populate mation (e.g., figures and formulas) in a separate window. this information into the pop-up conversion tool. Explainpaper is a novel AI-powered reading interface for In the following sections, we provide more details reading academic papers as well, ofering live explana- about our proposed features and plug-in marketplace tions to users upon highlighting sections of text and an for the next-gen document reader. interactive Q & A feature. However, these works are currently very limited in their features and scope. 3.1. Features
the following components as illustrated in Figure 1:
To begin the design process, we brainstormed features
that would be helpful to add to static document
readers such as Acrobat or Foxit, focusing on features that
can leverage NLP. During this stage, we surveyed the
literature and investigated existing plug-ins supported
by newer document viewers [
ument types, This process resulted in 26 potential feature
sugges• A set of more domain-specific features, and tions, which we narrowed down to 18 based on feasibility
• A centralized plug-in marketplace that would of implementation. These ideas were then categorized
allow users to further customize document read- by domain type, ultimately yielding 12 open-domain
ers with additional features.
3.1.1. Open-Domain Features
shows the former option, assuming that unit selection is
embedded inside of the plug-in pop-up. Allowing users to
select their unit of choice via the main document reader
toolbar (see Figure 5) would be another possibility.
When a single word is highlighted, document readers As with unit conversions, there could be a toolbar
opcould show users potential definitions of the term (Fig- tion at the top of document readers that would allow
ure 3), similar to [
Similarly, the long forms of acronyms/abbreviations could be shown to the user via pop-up tool tips (Figure 4). If the acronym is defined elsewhere in the paper, we could take a similar approach to [7, 8, 9] for extracting definitions; otherwise, retrieving it from online sources is also possible. A list of key definitions and acronyms could be included at the beginning or the end of the document as well, following [10]. The definition of math symbols could also be extracted from the text [11, 12].
Unit conversions could be either a case-by-case basis (i.e., users highlight specific numbers to convert like definitions/acronyms) or document-level (i.e., document reader automatically converts all units at once). Figure 2
Document readers could also provide users with the capability of directly copying tables from files like PDFs into Microsoft Word, Excel, Markdown, etc. to further manipulate, share or analyze. This table copying feature would reduce the need for hand transcribing and combining data from multiple tables in digital files. There could also be a selection hierarchy, allowing users to select specific parts of a table (e.g., a single cell, row, column, could be linked to an app/website for further action, as etc.) or the whole table itself. A version of this feature Google Chrome or Android currently does. The former is included in the Adobe PDF Extract API , but currently, is illustrated in Figure 7. Other ideas include linking tables can only be exported to CSV formats, so there is protein names to the Protein Data Bank for biology docroom for extension and it is still missing from all the PDF uments, linking references to their Google Scholar entry readers we surveyed. Similarly, an equation export- in scholarly articles, or linking ticker symbols to their Yaing plug-in could allow users to export math equations hoo Finance pages (e.g., finance.yahoo.com/quote/ADBE present in digital documents to their corresponding La- → ADBE) for finance documents.
TeX formulas so they are directly editable. Implementing this feature would be possible using image-to-latex algorithms [13, 14, 15].
Another way to enhance the document reading experience could be including a speed-reading plug-in that would allow allow users to customize the speed at which they read text in document readers, similar to Figure 7: Example address linkifying feature the service ofered by Spritz. Additionally, a sentiment analysis feature could allow users to assess sentiment at the document level and potentially at the sentence Similarly, document readers could link text to rellevel as well. Sentiment classification would be useful evant content. This is a trickier task than linkifying for a wide variety of document types, particularly when known entities, but the required extrapolation may be it is beneficial to understand a document’s tone/attitude. feasible in certain cases. For instance, if a file is identified Some approaches for document-level sentiment analysis as a restaurant menu, the document reader could link to have been proposed by [16, 17]. the corresponding Yelp or Google reviews page so users
For certain documents such as history books, scientific could see more pictures/reviews of diferent items (Figure papers, and poems, reading applications could also ofer 8). Or, if a movie title is identified inside a document, scholar notes. As an example, the document reader links to available movie times or streaming platforms could include a critic’s analysis of a text (e.g., in the could be generated. Another possibility would be searchsidebar) and/or their annotations throughout the file as ing selected keywords/phrases in a search engine or ecomments that the user could view. The notes could be commerce website (e.g., Amazon, Alibaba, etc.) to see distributed via a marketplace, and some of them could related products; Axesso Amazon API has implemented be set as paid access only if monetization is of inter- one such keyword search feature. Ultimately, this feature est. Users may be willing to pay to get access to the could be similar to how YouTube recommends products meta-information given by a scholar in the field to bet- based on the videos a user watches. ter understand the text itself, its historical context, the equations, potential errors, the author’s mindset at the time of the writing, and so on. A related feature idea is allowing users to leave shared comments in digital documents. For example, users could highlight a sentence or figure and then create a thread for further discussion (e.g., asking a question, ofering clarification, etc.), which Figure 8: Example restaurant linkifying feature could open up in a sidebar.
These human-in-the-loop features could help make up for the imperfections and the limitations of other Additionally, we could identify and create action AI-powered document plug-ins. However, the main chal- tasks that users could complete within the document lenges with implementing such features would be mod- reader itself. For example, if a date/deadline is identified erating/filtering the user content and respecting users’ in the text, users could be given the option to add it to privacy (e.g., we do not want a user to mistakenly post their calendar. Similarly, if a payment is mentioned, users their comments as public if they did not intend too). could have the option to pay directly inside of the document. Or, if there is language such as “You should notify 3.1.2. Domain-Specific Features X...” or “Please reach out to Y...” in a digital file, it might be helpful to give users the ability to send messages/eDocument readers could also incorporate domain- mails from the document reader as well. In general, these specific features such as linking text to known entities. tasks could be accessed via pop-up icons throughout the For example, addresses or business names could be au- document, but there could also be an overall list on the tomatically linked to Google Maps and phone numbers sidebar, for instance. 3.2. Plug-in Marketplace
In the previous sections, we describe many potential This work represents a preliminary, exploratory vision plug-ins for the next-gen document reader. However, the for the next-gen document reader. The next steps inaverage user will not need all these features when en- clude concretizing our ideas and assessing the viability gaging with digital files. Thus, to allow users to further of implementation. Specifically, we hope to conduct forcustomize their document reading experience and choose mal user studies to collect additional feedback on our which features they want to use, we propose the creation vision, further hone the proposed designs, and better of a centralized plug-in marketplace. This way, docu- understand which features would be most useful to endment readers could come with a few default plug-ins (e.g., users. Through these user studies, we may also generate definitions/acronyms or other open-domain features that additional ideas for potential document reader plugins. could be useful for most document types and users) and Further out in the future, we could also consider more users could add more via the marketplace at any time. complex features. For example, a filtering option would
Having a plug-in marketplace would also prevent doc- help readers focus on only the most relevant parts of the
ument readers from growing excessively in terms of size document, similar to the declutter feature from [
In this paper, we present our vision for the nextgen document reader that will transform static, isolated documents into connected, trustworthy, interactive sources of information. This vision includes 12 open-domain and 6 domain-specific features powered by NLP, which can be accessed by the user through contextual plug-in pop-up menus while reading digital files. To allow users to customize their reading experiences with document readers, we also propose a centralized plug-in marketplace inspired by modern IDEs and text editors. Next steps include conducting formal user studies to further hone our UI prototypes (github.com/catherinesyeh/nextgen-prototypes) and vision, while also considering additional complex features to improve user-document interaction such as filtering or question-answering. We hope this work inspires and excites others about the future of document readers. A. Head, K. Lo, Paper plain: Making medical tional Workshop on Semantic Evaluation (SemEvalresearch papers approachable to healthcare con- 2022), Association for Computational Linguissumers with natural language processing, arXiv tics, Seattle, United States, 2022, pp. 1671–1678. preprint arXiv:2203.00130 (2022). URL: https://aclanthology.org/2022.semeval-1.230. [3] R. Fok, A. Head, J. Bragg, K. Lo, M. A. Hearst, D. S. doi:10.18653/v1/2022.semeval- 1.230.
Weld, Scim: Intelligent faceted highlights for in- [12] V. D. Lai, A. P. B. Veyseh, F. Dernoncourt, T. H. teractive, multi-pass skimming of scientific papers, Nguyen, Symlink: A new dataset for scienarXiv preprint arXiv:2205.04561 (2022). tific symbol-description linking, arXiv preprint [4] A. Mostafavi, Sioyek, https://github.com/ahrm/ arXiv:2204.12070 (2022).
sioyek, 2021. [13] Y. Deng, A. Kanervisto, J. Ling, A. M. Rush, Image[5] A. Veyseh, F. Dernoncourt, D. Dou, T. Nguyen, A to-markup generation with coarse-to-fine attention, joint model for definition extraction with syntactic in: International Conference on Machine Learning, connection and semantic consistency, in: Proceed- PMLR, 2017, pp. 980–989. ings of the AAAI Conference on Artificial Intelli- [14] J.-W. Wu, F. Yin, Y.-M. Zhang, X.-Y. Zhang, C.-L. Liu, gence, volume 34, 2020, pp. 9098–9105. Image-to-markup generation via paired adversarial [6] S. Spala, N. Miller, F. Dernoncourt, C. Dockhorn, learning, in: M. Berlingerio, F. Bonchi, T. GärtSemEval-2020 task 6: Definition extraction from ner, N. Hurley, G. Ifrim (Eds.), Machine Learning free text with the DEFT corpus, in: Proceedings and Knowledge Discovery in Databases, Springer of the Fourteenth Workshop on Semantic Evalua- International Publishing, Cham, 2019, pp. 18–34. tion, International Committee for Computational [15] J. Zhang, J. Du, Y. Yang, Y.-Z. Song, S. Wei, L. Dai, A Linguistics, Barcelona (online), 2020, pp. 336–345. tree-structured decoder for image-to-markup genURL: https://aclanthology.org/2020.semeval-1.41. eration, in: International Conference on Machine doi:10.18653/v1/2020.semeval- 1.41. Learning, PMLR, 2020, pp. 11076–11085. [7] A. P. B. Veyseh, F. Dernoncourt, T. H. Nguyen, [16] T. Ito, K. Tsubouchi, H. Sakaji, T. Yamashita, W. Chang, L. A. Celi, Acronym identification and K. Izumi, Contextual sentiment neural network disambiguation shared tasks for scientific document for document sentiment analysis, Data Science and understanding, arXiv preprint arXiv:2012.11760 Engineering 5 (2020) 180–192.
(2020). [17] M. Rhanoui, M. Mikram, S. Yousfi, S. Barzali, A cnn[8] A. Pouran Ben Veyseh, F. Dernoncourt, Q. H. Tran, bilstm model for document-level sentiment analyT. H. Nguyen, What does this acronym mean? sis, Machine Learning and Knowledge Extraction 1 introducing a new dataset for acronym identifica- (2019) 832–847. tion and disambiguation, in: Proceedings of the [18] J. A. Teixeira da Silva, H. Bornemann-Cimenti, Why 28th International Conference on Computational do some retracted papers continue to be cited?, SciLinguistics, International Committee on Compu- entometrics 110 (2017) 365–370. tational Linguistics, Barcelona, Spain (Online), [19] M. J. Bolland, A. Grey, A. Avenell, Citation of re2020, pp. 3285–3301. URL: https://aclanthology. tracted publications: A challenging problem, Acorg/2020.coling-main.292. doi:10.18653/v1/2020. countability in Research 29 (2022) 18–25. coling- main.292. [20] A. Cohan, F. Dernoncourt, D. S. Kim, T. Bui, S. Kim, [9] A. P. B. Veyseh, N. Meister, F. Dernoncourt, T. H. W. Chang, N. Goharian, A discourse-aware atNguyen, Acronym extraction and acronym disam- tention model for abstractive summarization of biguation shared tasks at the scientific document long documents, in: Proceedings of the 2018 understanding workshop (2022). Conference of the North American Chapter of the [10] A. P. B. Veyseh, F. Dernoncourt, W. Chang, T. H. Association for Computational Linguistics: HuNguyen, MadDog: A web-based system for man Language Technologies, Volume 2 (Short Paacronym identification and disambiguation, in: pers), Association for Computational Linguistics, Proceedings of the 16th Conference of the Euro- New Orleans, Louisiana, 2018, pp. 615–621. URL: pean Chapter of the Association for Computational https://aclanthology.org/N18-2097. doi:10.18653/ Linguistics: System Demonstrations, Association v1/N18- 2097. for Computational Linguistics, 2021, pp. 160–167. [21] H. Lee, C. Park, S. Yoon, T. Bui, F. Dernoncourt, URL: https://aclanthology.org/2021.eacl-demos.20. J. Kim, K. Jung, Factual error correction for abdoi:10.18653/v1/2021.eacl- demos.20. stractive summaries using entity retrieval, arXiv [11] V. Lai, A. Pouran Ben Veyseh, F. Dernoncourt, preprint arXiv:2204.08263 (2022).
T. Nguyen, SemEval 2022 task 12: Symlink - [22] N. Rizvi, S. Gehrmann, L. Wang, F. Dernoncourt, linking mathematical symbols to their descrip- Margin call: an accessible web-based text viewer tions, in: Proceedings of the 16th Interna- with generated paragraph summaries in the mar