Named entity recognition (NER) is an important task in narration extraction. Narration, as a system of stories, provides insights into how events and characters in the stories develop over time. This paper proposes an architecture for NER on a corpus about social purpose organizations. This is the rst NER task speci cally targeted at social service entities. We show how this approach can be used for the sequencing of services and impacted clients with information extracted from unstructured text. The methodology outlines steps for extracting ontological representation of entities such as needs and satis ers and generating hypotheses to answer queries about impact models de ned by social purpose organizations. We evaluate the model on a corpus of social service descriptions with empirically calculated score.
of time (short term, long term) as a result of the organization’s activities.”1 Experts have developed
numerous Impact Models to help SPOs articulate the change they seek to achieve and how
that change is achieved [
In this paper, we describe our e orts in addressing the problem of matching client needs to SPO services. In order to match client needs, we must represent the services an SPO provides and how they satisfy needs, and the characteristics (e.g., age, gender, occupation) and needs of clients for whom they were designed. Although we have the means to represent an SPOs impact model, the information needed to instantiate each SPO’s model is buried in a variety of textual sources, such as service descriptions, client success stories, and eligibility criteria.
This paper presents an approach, based on Named Entity Recognition (NER), to extracting an
SPO’s impact model, i.e., “narrative”, from various text sources. NER is a crucial component in
understanding the narrative of a given text. By narrative, we mean a “system of stories structured
in such a way as to achieve a rhetorical purpose or vision” [
There are several challenges that we face when extracting terms that represent client needs and speci c resources that services provide. Consider the example “We welcome clients of all ages and o er services that bene t our core clients (young families, chronically homeless): mental health, education needs; community outreach.” that provides required information but is hard to parse. Firstly, there is a lack of language models for identifying social service client characteristics and needs, and how their services satisfy needs. Often, vague descriptors cause confusion about entities: what is a program, what is a service, what is the resource, what is the need. There are no standardized labels across SPO programs, services, clients, and eligibility criteria. Structured data about services is limited, while unstructured text describes various aspects of the service that are hard to infer, such as promoting services versus listing service details, or describing serviced communities versus listing client requirements. Finally, information related to the scheduling or expected sequence of services is often incomplete or unknown, such as quality, service capacity, or availability.
Named Entity Recognition (NER) is a method for identifying the types of terms in unstructured
text. Common terms include a person, organization, place, date, currency, and numbers [
There is a number of pre-trained language models capable of named entity recognition.
Each one is trained on either a specialized or general dataset. Schmitt et al. [
In this section, we highlight several approaches and their methods that identify entities in the
text and can assist in building an SPO’s impact model narrative. Linguistic properties alone
provide a great deal of structure to the text being analyzed. For example, Chiarello et al. [
Statistical models rely on data-driven algorithms and encompass both frequency-based and probability-based models [23, 24] for ranking found entities [25], group generalization [17, 19], and calculating similarity scores between documents [20]. Machine learning methods include deep learning architectures for NER tasks [26]. Several have characteristics useful for narrative extraction, such as temporal factors, rules, or linguistic properties, and utilize methods such as BiLSTMs [27, 28, 29, 30], ELMo [28, 31, 32, 33], and BERT [34, 35, 36, 37, 38, 39].
This section summarizes our methodology for performing the NER task in the SPO domain. Our proposed NER architecture is depicted in Figure 1. The input is a corpus of unstructured text
D documents Service descriptions found online POS Tagger Dependency
Tree Coreference
Resolution Semantic Roles [program]-offers-[service] [service]-delivers-[satisfier] [satisfier]-satisfies-[need] [service]-eligibleFor-[demographic] [service]-requires-[constraint]
Apply RT rules
T triples
T+Coreference
Resolution Common Impact Data Standard
T+Conjunction
Resolution Apply T Chain
Rules
Apply RE Rules Semantic Roles
Parser
Entity 1
Entity 2 …
Entity n
Matthews correlation coe icient (MCC) score for rule riE .
describing SPOs. It incorporates the Common Impact Data Standard [
0, wi = riE (Td) ⇥ e if mcci > 0 otherwise
mcci we =
P wei for all rules riE 2 rE that extract entity e. |rE | (1) (2) (3)
Our method begins by relying on a Stanford NLPCore parser [40] to generate a set of linguistic properties about the service descriptions. We use its part-of-speech (POS) tagger to identify nouns, verbs, adjectives, and so on. Second, the parser creates a dependency tree identifying word modi ers, conjunctions, as well as subjects, predicates, and objects. Third, the parser generates coreference resolutions between terms, associating pronouns like “they” and “our” with the nouns or proper nouns they refer to. Accuracy and further processing is limited by the accuracy of the dependency trees and coreference resolutions generated by the NLPCore parser.
Once the parser has annotated the text with linguistic properties, custom rules rxT 2 RT combine key dependencies to form subject-predicate-object triples tx 2 T . In some literature, the triple relation is referred to as subject-verb-object (SVO), but T -triples represent a broader structure that does not rely on verbs as predicates alone. Each triple contains three slots: a subject (s), a predicate (p), and an object (o), forming the structure:
tx = { s(“subject”), p(“predicate”), o(“object”) } .
For example, consider the sentence “St. Mary’s provides education services.” Here we see that “St. Mary’s” is the subject, “provides” is the predicate, and “education services” is the object. Consider a rule riT where, given the three terms A, B, and C, and dependencies (nsubj, obj, obl) If a nsubj dependency exists between B and A, an obj dependency exists between B and C, an obl dependency does not exist between B any other term,
Then tx = {s(A), p(B), o(C)}.
By applying this rule to the sentence above, we can infer the T triple:
tx = { s(“St Mary’s”), p(provides), o(“education services”) }.
While this example rule is easily inferred from the dependencies alone, 18 rules riT 2 RT have been empirically identi ed to extract subject-predicate-object relationships.
Next, each T -slot is extended with their coreference and conjunction terms, if any, using a depth- rst search. For example, in the sentences “St. Mary’s provides education services. They also prepare hot meals.”, the pronoun “They” refers to the proper noun “St. Mary’s”. Hence we infer that in addition to “education services”, “St Mary’s” also provides “hot meals”, giving: t1 = { s(“St Mary’s”), p(“provides”), o(“education services”) }
t2 = { s(“St Mary’s”), p(“prepares”), o(“hot meals”) }
Next, we resolve conjunctions with terms in each T -slot. Conjunctions are lists of terms connected by terms like a comma, “and” and “or”. For example, given the sentence “St. Mary’s provides education services, a soup kitchen, and religious counselling.”, we see that all terms following “provides” are of the same type, a “need satis er.”
Like subjects and objects, the predicate can also be a conjunction. For example, in the sentence “St. Mary’s provides education services and prepares hot meals.”, “provides” and “prepares” are both verbs connected as a conjunction in the dependency tree, and hence both have “St Mary’s” as their subject. However, they each have their own object, producing two T -triples, namely t1 = { s(“St Mary’s”), p(“provides”), o(“education services”) }
t2 = { s(“St Mary’s”), p(“prepares”), o(“hot meals”) } To ensure we capture all combinations of T -triples, our model uses a depth- rst search to generate hypotheses for all combinations of connected subjects, predicates, and objects.
Given a list of T -triples, and coreferences and conjunctions resolved, we build a chain of T triples that provide additional structure to the terms in the text. The rules simply connect object slot (X) values in one t1 triple to subject slot s(X) values in another t2 triple, If t1 = {s(A), p(B), o(C)} and t2 = {s(C), p(D), o(E)} Then [{s(A), p(B), o(C)}; {s(C), p(D), o(E)}] is a T -chain and
t3 = {s(A), p(D), o(E)}.
In the sentence “St. Mary’s provides education services to adult learners.” we see two T -triples: t1 = { s(“St. Mary’s”), p(“provides”), o(“education services”) }
t2 = { s(“education services”), p(“to”), o(“adult learners”) } Chaining them together with the rule above using the “education services” term, we can infer that “St Mary’s” o ers services to adult learners, generating a new T triple: t3 = { s(“St. Mary’s”), p(“to”), o(“adult learners”) }
From T -triples and T -chains, we can apply additional rules riE 2 RE to extract named entity types. For example, we see that “St Mary’s” is the program, “education services” is the need satis er, and “adult learners” are the clients. The rules utilize all available information about the text, including POS tags, dependencies, and their T -slots. For example, given terms A, B, C: If tx = {s(A), p(B), o(C)}, where A is a proper noun, B is a synonym for “o ers”,
B is a 3rd person singular present verb, and C is a plural noun
Then A is a program and C is a need satis er.
Here, synonyms for “o ers” have been empirically identi ed as keywords used by services
providers to describe what need satis ers they o er to clients, and include the terms
"provides", "o ers", "o er", "provide", "provided", "o ered", and "o ering". Similar extractions can
be performed for additional semantics de ned by Common Impact Data Standard [
The evaluation of our model is based on the performance of each rule, aggregated by entity type into a single entity score, namely we. The data contains information about SPOs, provided by Help Seeker Technologies (https://helpseeker.co). The testing data consists of 16,048 documents d 2 D that contain SPO descriptions. Of those, 7,359 documents had a total of 76,592 unique T -triples extracted. Constructed from the triples, there were 147,299 T -chains found in 6,260 documents. Of those documents that had T -triples, 4,860 descriptions had at least one term extracted. In total 366,588 terms were extracted, and of those 48,729 were assigned an entity type using rules in RE .
To evaluate the model, a number of documents were selected randomly for each entity, and the extracted terms were manually analyzed. Table 3 lists the results of each rule riE identifying an entity e. The rule’s label indicates its number and which slot was used (e.g. Rule = “o-45” means the “o” slot for rule 45). All rules rely on T -triples. Those marked with (]) also rely on T -chains. Each entity e extracted from a document’s triples Td was classi ed as correct (1) or incorrect (0), as per Equation 1.
We note that not all entities have the same number of rules and not all entities are covered equally. For example, the Need Satis er entity has the largest coverage with 13 rules while Client Characteristics, Desired States, and Required Criteria only have one. We also note that the MCC score is sensitive to large discrepancies between true (TP,TN) and false (FP,FN) values. Rules that identify signi cantly more true values but are not good at excluding false values can produce a negative MCC score, despite a high F-score, as marked by (§), and include Client Characteristic and Required Criteria.
The model’s performance is evaluated by its aggregate score for each entity, namely we, as per Equation 3. The evaluation uses the ROC-AUC score to determine whether a high we weight correlates with correct classi cation. The results are listed in the AUC column of Table 3. Any AU C 0.7 is considered acceptable, and marked by ([).
Based on the we score, the model has good performance on extracting the “Client Description”, “Need Satis er”, “Need Satis er Description” and “Program Name” entities. The model also performs well with a high F-score on the “Client Characteristic” and “Requirement Criteria” entities but resulted in a low we due to a negative MCC score. In cases where the MCC score was negative with a high F-score, we point out that if accuracy metrics (precision and recall) for the rule are high, the rule performed well on NER tasks but is limited to true positives only.
Relying on the entities that were correctly extracted, we can construct a set of semantic roles to build SPO narratives. For example, consider a particular program that delivers language classes (need satis er) to new immigrants (a client characteristic). We can specify what requirements these clients must meet before receiving these satis ers, such as language skill assessments. Knowing that another program o ers language skill assessments, we can connect the two programs, de ning a chain of SPO programs. Rule statistics and score based on MCC, and an aggregate model score we evaluated by AUC. ] a rule based on T-chains § a high F-score > 0.7 [acceptable AU C 0.7 value for a given we model score.
In this paper, we propose a model for extracting SPO-related entities from descriptions. We also present the challenges and state of the NLP
eld, namely its lack of SPO-related corpora and
pre-trained language models. Our model relies on our previous work for representing social
services entities and semantics, namely the Common Impact Data Standard ontology [
In future work, the model will be extended with additional features and training data. Negated phrases will generate semantics that negate a relationship, such as “does not o er”. A larger corpus with correct annotations will allow for better scoring methods, more rules, the use of statistical methods, and the training of supervised machine learning models. Finally, by incorporating available data associated with speci c entities and extracted SPO narratives, we could perform analysis on an SPO’s performance and suitability at a given time. For example, we cloud track a client’s development as they transition from one program to another, based on the paths they take, the need satis ers they qualify for, and ultimately use. extraction and visualization of narratives, in: Fourth International Workshop on Narrative Extraction from Texts, at 43rd European Conference on Information Retrieval, volume 2860, Luca, Itally, 2021, pp. 33–40. URL: https://btracker.host.ualr.edu. [16] E. Sciore, Query Processing, in: O. Curé, G. Blin (Eds.), RDF Database Systems, Morgan Kaufmann, Boston, 2015, pp. 145–167. URL: https://www.sciencedirect.com/science/article/ pii/B9780127999579000067. doi:https://doi.org/10.1016/B978-0-12-799957-9. 00006-7. [17] K. Balog, M. Bron, M. De Rijke, Category-based query modeling for entity search, in: Lecture Notes in Computer Science (including subseries Lecture Notes in Arti cial Intelligence and Lecture Notes in Bioinformatics), volume 5993 LNCS, 2010, pp. 319–331. doi:10.1007/978-3-642-12275-0_29. [18] N. Craswell, G. Demartini, J. Gaugaz, T. Iofciu, L3S at INEX 2008: Retrieving entities using structured information, Lecture Notes in Computer Science (including subseries Lecture Notes in Arti cial Intelligence and Lecture Notes in Bioinformatics) 5631 LNCS (2009) 253–263. doi:10.1007/978-3-642-03761-0_26. [19] K. Balog, M. Bron, M. De Rijke, Query modeling for entity search based on terms, categories, and examples, ACM Transactions on Information Systems 29 (2011). doi:10.1145/ 2037661.2037667. [20] D. Garigliotti, K. Balog, On type-aware entity retrieval, in: ICTIR 2017 - Proceedings of the 2017 ACM SIGIR International Conference on the Theory of Information Retrieval, Association for Computing Machinery, Inc, 2017, pp. 27–34. doi:10.1145/3121050.3121054. arXiv:1708.08291. [21] P. Quaresma, V. Beires Nogueira, K. Raiyani, R. Bayot, T. Gonçalves, From Textual Information Sources to Linked Data in the Agatha Project, in: Lecture Notes in Computer Science (including subseries Lecture Notes in Arti cial Intelligence and Lecture Notes in Bioinformatics), volume 12057 LNAI, 2020, pp. 79–88. URL: http://arxiv.org/abs/1909.05359. doi:10.1007/978-3-030-46714-2_5. arXiv:1909.05359. [22] M. C. McCord, J. W. Murdock, B. K. Boguraev, Deep parsing in Watson, IBM Journal of
Research and Development 56 (2012) 1–15. doi:10.1147/JRD.2012.2185409. [23] L. Hong, B. D. Davison, Empirical study of topic modeling in twitter, in: Proceedings of the rst workshop on social media analytics, 2010, pp. 80–88. [24] S. Robertson, H. Zaragoza, The probabilistic relevance framework: BM25 and beyond, volume 3, 2009. doi:10.1561/1500000019. [25] P. H. Oza, L. Dietz, Which entities are relevant for the story?, in: CEUR Workshop
Proceedings, volume 2860, 2021, pp. 41–48. URL: http://ceur-ws.org/. [26] J. Li, A. Sun, J. Han, C. Li, A Survey on Deep Learning for Named Entity Recognition, IEEE Transactions on Knowledge and Data Engineering 34 (2022) 50–70. doi:10.1109/ TKDE.2020.2981314. arXiv:1812.09449. [27] B. Taillé, V. Guigue, P. Gallinari, Contextualized embeddings in named-entity recognition: An empirical study on generalization, Lecture Notes in Computer Science (including subseries Lecture Notes in Arti cial Intelligence and Lecture Notes in Bioinformatics) 12036 LNCS (2020) 383–391. doi:10.1007/978-3-030-45442-5_48. arXiv:2001.08053. [28] M. A , C. Latiri, BE-BLC: BERT-ELMO-based deep neural network architecture for English named entity recognition task, Procedia Computer Science 192 (2021) 168–181.
URL: https://doi.org/10.1016/j.procs.2021.08.018. doi:10.1016/j.procs.2021.08.018. [29] G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer, Neural architectures for named entity recognition, 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2016 - Proceedings of the Conference (2016) 260–270. doi:10.18653/v1/n16-1030. arXiv:1603.01360. [30] Z. Jie, W. Lu, Dependency-guided LSTM-CRF for named entity recognition, EMNLPIJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference (2019) 3862–3872. doi:10.18653/v1/d19-1399. arXiv:1909.10148. [31] Y. Peng, S. Yan, Z. Lu, Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets, BioNLP 2019 - SIGBioMed Workshop on Biomedical Natural Language Processing, Proceedings of the 18th BioNLP Workshop and Shared Task (2019) 58–65. doi:10.18653/v1/w19-5006. arXiv:1906.05474. [32] M. Ul ar, M. Robnik- ikonja, Cross-lingual alignments of ELMo contextual embeddings (2021) 1–30. URL: http://arxiv.org/abs/2106.15986. arXiv:2106.15986. [33] C. Dogan, A. Dutra, A. Gara, A. Gemma, L. Shi, M. Sigamani, E. Walters, Fine-Grained Named Entity Recognition using ELMo and Wikidata (2019). URL: http://arxiv.org/abs/ 1904.10503. arXiv:1904.10503. [34] T. Moon, P. Awasthy, J. Ni, R. Florian, Towards Lingua Franca Named Entity Recognition with BERT (2019). URL: http://arxiv.org/abs/1912.01389. arXiv:1912.01389. [35] F. Souza, R. Nogueira, R. Lotufo, Portuguese Named Entity Recognition using BERT-CRF (2019). URL: http://arxiv.org/abs/1909.10649. arXiv:1909.10649. [36] C. Liang, Y. Yu, H. Jiang, S. Er, R. Wang, T. Zhao, C. Zhang, BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision, Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2020) 1054–1064. doi:10.1145/3394486.3403149. arXiv:2006.15509. [37] S. Zhou, J. Liu, X. Zhong, W. Zhao, Named Entity Recognition Using BERT with Whole World Masking in Cybersecurity Domain, 2021 IEEE 6th International Conference on Big Data Analytics, ICBDA 2021 (2021) 316–320. doi:10.1109/ICBDA51983.2021.9403180. [38] P. Röttger, J. Pierrehumbert, P. Rottger, J. Pierrehumbert, Temporal Adaptation of {BERT} and Performance on Downstream Document Classi cation: Insights from Social Media, in: Findings of the Association for Computational Linguistics: EMNLP 2021, Association for Computational Linguistics, Punta Cana, Dominican Republic, 2021, pp. 2400– 2412. URL: https://aclanthology.org/2021. ndings-emnlp.206. doi:10.18653/v1/2021. findings-emnlp.206. arXiv:2104.08116. [39] K. Vani, S. Mellace, A. Antonucci, Temporal embeddings and transformer models for narrative text understanding, CEUR Workshop Proceedings 2593 (2020) 71–77. arXiv:2003.08811. [40] C. D. Manning, J. Bauer, J. Finkel, S. J. Bethard, The Stanford CoreNLP Natural Language Processing Toolkit, Aclweb.Org (2014) 55–60. URL: http://macopolo.cn/mkpl/products.asp.