Bharathi-An Applied Semantic Intelligence Use Case for Public Data in India

Abstract

Data sets published for public consumption related to governance typically contain common metadata. This metadata normally describes the publishing organisation, the domain, the administrative region and a topic that the data best describes. A relevant semantic vocabulary will not only facilitate linking the metadata through meaningful contexts but also enable grounding related information from diverse public data sets using semantic entities defining the metadata. In this paper, we present Bharathi -Linked Data Vocabulary for the Indian context. Bharathi contains information regarding government organisations at the union & state government level, administrative regions, sectors, sub sectors and common topics used frequently in the vocabulary of the government functions. Further, Bharathi contains links to other open vocabularies such as GeoNames for geographical locations. The schema of Bharathi uses existing established ontologies making it inter-operable and extensible. We describe Bharathi along with a live use case of its application in accomplishing a semantic Covid19 data analysis for India.

Keywords

• eol>Linked Data
• Knowledge Graph
• Ontology
• Semantic Intelligence

1. Introduction

discovery of inter-connected relationships sample information from Bharathi. across these pivotal entities by humans and We start with the canvas of Related Work machines. In many cases, the same entity (a (Section 2) where we list references to simgovernment organisation or an administra- ilar work and set the context for our contive region) is referred by its respective code tribution. Next, we provide an overview of across diferent web sites making it a humon- Bharathi followed by an introduction to the gous task to connect the information across Bharathi ontology in Section 4. Section 5 dethe two data sources although they relate to tails the process of generation of Bharathi. the same entity. Therefore, creating an ap- Real world applications of Bharathi are prepropriate semantic framework for a knowl- sented in Section 6. We conclude with a brief edge based representation for this metadata summary and thoughts on future directions is paramount. Such a semantically enriched in Section 7. knowledge representation will substantially help in linking public data sets and also provide a valuable inter-operable, reproducible 2. Related Work and machine-readable resource for the scientific and industrial community. To the best of our knowledge, no reasonably large and up-to-date corpus of linked data information about government entities, their classifications and their inter relationships specifically for the Indian context has been made publicly available yet, in the form of a machine readable resource. This paper presents a use case for applied semantic intelligence – Bharathi and its potential applications.

Bharathi is populated using a semiautomated pipeline that periodically harvests data from various government of India websites, annotating the entities and lifting the semantics into a RDF6 grounded by a sound model based on established vocabularies such as Organisation Ontology7, SKOS Vocabulary8 and the GeoNames Knowledge base9. We provide a web interface accessible at http://semanticwebindia.in/bharathi for exploring the underlying knowledge graph of Bharathi, enabling users from outside the scientific community to find, retrieve and Several research eforts such as [ 1, 2, 3, 4, 5, 6, 7, 8 ] have shown interest in linking public data with semantic metadata to enable intuitive linking of information from diverse sources. The research eforts specifically to model the metadata around public data can be found in [ 9 ] where the authors presents Gridworks-based data workbench application to help in converting open government data to linked data by using standard Data Catalog Vocabulary. The data catalogs of open data are an input to the application, yielding the linked data form. [ 10 ] presents a semantic government vocabulary to create annotations of open government data. It details the diferent layers of the vocabulary depending on diferent ontological terms found in open government datasets. It also demonstrates how to use the vocabulary to annotate open government data at diferent levels. [ 11 ] explores the documentation needs for open government data with specific focus on metadata interoperability issues. It presents two methodologies for interoperability after studying a variety of open govthe Registrar General and Census Commissioner, India

6Resource Description Framework: https://www. ernment metadata, globally recognized stanw3.org/RDF/ dards and guidelines. The outcomes of the 7org: http://www.w3.org/ns/org# ifrst approach are juxtaposed with those of 98sgkno: sh:thtptt:p//:w//wwww.wge.won3.aomrge/s2.o0r0g4/02/skos/core# the second in the light of interoperability during the metadata integration process. [ 12 ] metadata specifically for the Indian context presents a methodology, five basic steps and a to enable a powerful semantic integration of model to publish statistical data coming from public data related to governance in India. tabular data sources or relational databases in the form of linked open data. It follows the best practices for publishing linked data, 3. Bharathi Overview a W3C working group note such as providing URI construction suggestions following Bharathi is populated using data extracted URI policy rules, reuse of standard vocab- from a number of government of India web ularies, such as the Data Cube Vocabulary, sites whose content is created and mainSKOS, etc., converting data to RDF and pro- tained by the National Informatics Centre viding access to the converted data. [ 13 ] de- (NIC), Ministry of Electronics and Informascribes the attempt at building “AKTivePSI”- tion Technology, Government of India. In the an initiative by the Ofice of Public Sector In- current version of Bharathi, we consider information, UK to adopt semantic web tech- formation from https://lgdirectory.gov.in/13, nology for large scale integration, sharing goidirectory.nic.in/, http://censusindia.gov. and reusing the public data for the benefits in/ and https://data.gov.in/ to build the of government, businesses and citizens alike. knowledge graph for metadata around adThe key outcome due to the success of this ministrative regions and their codes, governpilot project led to more increased aware- ment entities and their classification, various ness amongst the government bodies about sectors and their links within the respective the power of semantic web technology. government function. While Bharathi is

While there is a prolific attempt to intro- extensible in including new sources of induce a semantic knowledge framework to formation, the extraction pipeline will need extract and maintain the metadata required modification depending on the structural for semantic integration of public data in specifications of the source. The output the western nations, there have been none from the extraction pipeline is converted such coordinated and specific eforts in In- into semantic entities using the Bharathi dia to the best of our knowledge. DBpedia ontology (Refer Section 4). The Bharathi resources (extracted from Wikipedia infor- knowledge graph is currently accessible mation) do exist for India, though in many from a Virtuoso triplestore with a SPARQL cases they are not up-to-date with the in- endpoint at http://www.semanticwebindia. formation published on the Government of in/BharathiLive/sparql. All entities of India web sites. The motivation to create Bharathi are semantically linked and accesBharathi stems from extensive research work sible using persistent resolvable identifiers that accomplished semantic integration of following the W3C best practices. Bharathi open data tables published in data.gov.in us- is released under a Creative Commons ing entities from Linked Open Data (LOD)10 Attribution-ShareAlike 4.0 International namely DBpedia11 and Wikidata12 [ 14 ], [ 15 ]. License14. The web interface is accessible at Bharathi is our contribution to create, main- http://www.semanticwebindia.in/bharathi/ tain and sustain this knowledge graph of allowing users to search and browse 10LOD: https://lod-cloud.net/ 13https://lgdirectory.gov.in/ : LOCAL GOVERN11DBpedia: https://wiki.dbpedia.org/ MENT DIRECTORY – Complete directory of land re12Wikidata: https://www.wikidata.org/wiki/ gions/revenue, rural and urban local governments Wikidata:Main_Page 14https://creativecommons.org/licenses/by-sa/4.0/ Bharathi. The knowledge graph is up- tion of established vocabularies was made to dated every 3 - 6 months, depending on ensure 1) interoperability of the knowledge the changes to the information at source. base through reuse 2) adherence to global Information related to the knowledge graph definitions of schema for known entities by dumps and ontology metrics can be accessed reusing terms and most importantly 3) leverat http://semanticwebindia.in/bharathi/ aging of community support for linked open BharathiKG. data cloud vocabularies.

The core element of Bharathi ontology is the Organisation class 4. Bharathi Ontology (bhorg:Organisation16) derived as a subClass of org:organization. This entity is used to The underlying schema for Bharathi uses es- denote any public, private or government tablished ontologies and vocabularies such organsiation in India. The other core eleas the Organisation ontology and the SKOS ment of the Bharathi ontology is the use vocabulary to define the building blocks for of skos:Concept to represent the classificathe metadata pertaining to government or- tion hierarchy to denote one of the three ganisations, administrative regions, gover- classifications namely –an organisation nance sectors and the classification hierar- classification, a governance sector or a topic chy linking these entities. Figure 1 illustrates within a sector. Separate Concept Schemes the domain model for the Bharathi ontol- (skos:ConceptScheme) have been defined to ogy. The Bharathi ontology with name prefix distinguish between these three diferent bhorg15 contains the base classes and proper- types of classifications. Each concept scheme ties used by the knowledge graph. The selec- has its own hierarchy starting from a Top 15bhorg: http://semanticwebindia.in/bharathi/ont/ org/ 16bhorg:Organisation - http://semanticwebindia.in/ bharathi/ont/org/Organisation Concept (skos:hasTopConcept). An instance of type bhorg:Organisation has the following object properties 1) org:hasSubOrganization –inherited from the org:Organization class and used to link sub organisations 2) org:subOrganizationOf –inherited from the org:Organization class and used to link to the parent organisation 3) org:linkedTo –inherited from the org:Organization class and used to link related organisations and 4) bhorg:hasSpatial –property defined in the extended class to link to the GeoNames Feature (gn:Feature) associated with the instance. The bhorg:Organisation entity also has data properties to associate the names (primary and alternate), URL, abbreviations and codes with the organisation. All the extended classes and properties of the Bharathi ontology have been provided with detailed annotations in line with RDF principles.

Refer Figure 1, the oval objects represent entities and the corresponding rectangular entities linked by rdf:type (a) labelled link (black dashed link in the figure) indicate the respective instances. All government organisations and administrative regions are instances of the bhorg:Organisation entity while organisation classifications, governance sectors and their hierarchical structures are defined using the SKOS ontology.

As illustrated in Figure 1, “Science and Technology Department West Bengal” is a government entity having classification “State Department” and is a sub organisation in the “State” of “West Bengal”. The administrative region “West Bengal” is linked to a GeoNames Feature using the corresponding GeoNames knowledge base. While the SKOS concepts “State” and “State Department” belong to the concept scheme “Organisation Classification”, the SKOS concept “Science Technology & Research” belongs to the concept scheme “Governance Sector” and is associated with the organisation “Science and Technology Department West Bengal”.

5. Generating Bharathi

Bharathi is built using a semi-automated ex

traction pipeline that periodically crawls the Government of India web sites mentioned in Section 3, annotates the semantic entities and serializes the knowledge graph tuples in the form of a N-Triples file 17. The knowledge graph is generated inline with the Bharathi model explained in Section 4. Each of these steps is explained in detail below: 5.1. Extraction of Metadata

The source files exists in the form of either

HTML pages or CSV (Comma Separated Values) files. The extractor is customised for each of the source web sites to take care of the specific nuances and organisation of information within the respective source web site. Note that this feature of the extraction pipeline can be extended in the future to include more sources of web sites as identified.

The information extracted mainly consists of the government entity, its descriptors (such as name, URL, abbreviations), the classification of the organisation and the list of sub organisations. This information is consolidated from various sources into a large single data set (CSV file). The information regarding sector, sub sector and related topics is extracted into a separate CSV dataset as this data structure is diferent from the extraction format used for the government organisations.

The administrative regions, sectors, topics and other organisations do change periodically (name changes, addition of new districts etc.). These changes are handled using the Provenance Ontology18. Wherever the source mentions a version, the same is recorded in Bharathi with a history of activity that triggered the change. URL links 17https://www.w3.org/TR/n-triples/ 18PROV-O: The PROV Ontology –https://www.w3. org/TR/prov-o/ generated for the Bharathi entities are static. used to create and serialize the tuples usThey don’t change for the entities once de- ing the Bharathi ontology definitions disifned in Bharathi thus assuring continuity to cussed in Section 4. Unique persistent URLs the users of Bharathi. are generated for each instance following the RDF principles. The knowledge graph 5.2. Entity Annotations can be accessed at http://semanticwebindia. in/bharathi/BharathiKG. Figure 2 illustrates A unique entity is created for each new in- the overall architecture for the generation of stance of a government organisation or a the Bharathi knowledge graph. classification depending upon the attributes of the entity extracted. GeoNames web services are used to link administrative regions 6. Real World Applications to its corresponding GeoNames feature. The using Bharathi entity annotation routine is sensitive enough to detect that entities with the same label We present two real world applications using (hasName or prefLabel) need not represent the Bharathi knowledge graph. the same entity in the Bharathi knowledge graph. For eg. “Chandigarh” is a Union Ter- 6.1. Web Interface to Bharathi ritory, a District as well as a Smart City in India. To distinguish this aspect, surrounding This is a web application interface for users attributes of the extracted information from outside the scientific community to search, the source web site are used to make the dis- browse and explore the entities and their retinction. We use three versions of the Entity lationships in Bharathi. The users are alAnnotator, one each for government organi- lowed to search for a particular entity or find sations & administrative regions, one for or- it through one of its classifications. The inganisation classifications and one for gover- formation is organised in a hierarchical mannance sectors and topics. ner to understand the hierarchical classification of each entity in the government func5.3. Serialization of the tion. Further, the linked entities can be traKnowledge Graph versed just as reaching out from one link to the other in a web page. Each page regardWe have created a Python 3.6 script to ing a government organisation or an admingenerate the N-Triples (.nt) file using the istrative region or a classification presents a CSV file generated in the extraction step. consolidated identity to the entity. This is SPARQL wrapper rdflib routines have been an important feature of Bharathi. Most of the metadata used for the Indian context is “Chandigarh - Smart City”. The right pane spread across many sources with the same of the functional page shows the hierarchy administrative region sometimes referred by of the selected entity. its code or its various names or its abbreviation. 6.2. Semantic Covid19 India

Further, provenance history is Analysis available for public view at http: //semanticwebindia.in/bharathi/download. This application has been built by harnessAlso, version history for a semantic entity in ing the semantic framework of Bharathi. Bharathi can be viewed at its functional link Covid 19 datasets are sourced from multie.g. (http://semanticwebindia.in/bharathi/ ple public data sources such as https://www. home?id=14900&type=org). A feedback covid19india.org/ for covid data related to Inpage has been provisioned to capture the dia and https://datahub.io/core/covid-19 for user’s inputs on how the information can covid data related to other countries. A debe enhanced with additional metadata from tailed entity disambiguation routine identiother sources. Figure 3 shows the landing ifes the linkable semantic entities from the page of Bharathi for the external user. Users raw Covid19 datasets. The Covid19 indicacan search for an administrative region, tors such as “Total Confimed”, “Total Degovernment organisation or a sector / sub ceased”, “Total Recovered”, “Total Tested” sector. Bharathi also holds a modest set of etc. were created as sub topics under the common topics found in public data sites Sector - “Health”, “Sub Sector” - "Covid19". published by the government. Figure 4 Each dataset refers to a geographical locashows the functional page for the entity tion for which the indicators have been collected and these could be at the state or the erarchy of the administrative regions, with sub state Level (eg. districts, cities). These ge- minimal efort, the Covid19 application facilographical entities are the administrative re- itates aggregation of the various indicators gions already existing in Bharathi. Using the at the state level (all districts within a state semantic links of the administrative regions have a hierarchical relation to their parent with GeoNames resources and their classi- state) and analysis of Covid19 transmission ifcations such as “State” or “District” in In- across states sharing boundaries. The DBpedia, we are able to answer complex intuitive dia knowledge graph is linked to the GeoNCovid19 Analysis questions such as 1) What ames knowledge base and the GeoNames enis the trend of “Total Confirmed Cases” in tities are linked to Bharathi administrative the neighbourhood of “District” Coimbatore regions. Therefore, this semantic link en(a district in the state of Tamil Nadu in India) ables us to explore the efect of Covid transwithin a distance of 100 km. 2) How do states mission across states sharing boundaries. with comparable population density fare on Figure 5 and Figure 6 show a the “Number of Deaths” in a given time pe- glimpse of Covid19 Analysis India riod? 3) What are the indicators in the family application and can be accessed at of “Testing Indicators” that can be compared http://sandhicovid.semanticwebindia.com/. for a given region and timeline? Each of the above questions is answered by linking the Analysing Covid19 events : Conceptuadditional metadata from related knowledge ally an event brings together an activity or graphs using the linked data properties of the a milestone, a location and a time period in a Bharathi knowledge graph. single entity. We extended the “Event Class” Additionally, using the classification hifrom the Event ontology19 to define a cus- data available in the Bharathi knowledge tom “Event” in the Sandhi Covid19 applica- graph. Thus Bharathi enables advanced analtion. Apart from the usual attributes inher- ysis based on entity-centric search, exploited from the standard event ontology, we in- ration and information discovery through troduced object properties for “predecessor” the linked data properties of the underlying and ”successor”, thus enabling an in-depth knowledge graph. analysis of related events in the Covid19 timeline. “Lockdown 1.0”, “Lockdown 2.0” 6.3. Sample SPARQL queries to etc. were created as individual instances the Bharathi Knowledge of the extended Event class. The semantics facilitated by these linked entities enable Graph a detailed “Event Based Analysis” using the Here we present an assorted list of SPARQL Covid19 data. queries to the knowledge graph and describe

Figure 7 shows “Event Based Analysis” their expected result. feature of Covid19 Analysis India applica- The SPARQL query in Table 1 returns all tion. the top level organisation classifications in

Note that this semantic layer can be ex- the Government of India. tended to any collection of open datasets The SPARQL query in Table 2 returns containing entities pertaining to the Indian all the States and Union Territories of Incontext that can be linked to the meta- dia along with their lat-long coordinates by linking Bharathi knowledge graph with the 19The Event Ontology: http://motools.sourceforge. GeoNames knowledge base. net/event/event.html

Knowledge Graph – Compare a Neighbourhood

the government organisations and their sub organisations belonging to the classification - "Legislature of India".

Union Territories of India along with their lat-long coordinates by linking Bharathi knowledge graph with the GeoNames knowledge base

PREFIX gn: <http://www.geonames.org/ontology#> PREFIX skos: <http://www.w3.org/2004/02/skos/core#> PREFIX bhorg: <http://semanticwebindia.in/bharathi/ont/org/> PREFIX org: <http://www.w3.org/ns/org#> PREFIX wgs84_pos: <http://www.w3.org/2003/01/geo/wgs84_pos#> select ?org1 ?gc ?classname ?name ?lat ?long where { ?org1 a bhorg:Organisation. ?org1 bhorg:hasName ?name. ?org1 bhorg:hasSpatial ?gc. ?org1 org:classification ?c. ?c skos:prefLabel ?classname. FILTER (?classname IN ("State", "Union Territory", "National Capital Territory")). ?gc wgs84_pos:lat ?lat. ?gc wgs84_pos:long ?long } order by ?name

7. Conclusions and Future

Work

Comparisons with DBpedia : DBpedia We have introduced Bharathi - Linked Data is project to convert structured content of Vocabulary for the Indian context, which fa- Wikipedia into linked data and does contain a lot of information regarding India documented in Wikipedia. However, this information is largely incomplete and does not align with some of the most credible updates published by the Government of India (GoI) websites. Bharathi is an efort to exclusively extract information from GoI websites and maintain its currency.

There are several limitations of the Bharathi knowledge graph that are the focus of ongoing and short-term eforts.

Bharathi can be enriched with reasoning statements from the OWL vocabulary to enable reasoning and creation of new facts using reasoners such as Apache Jena. We also intend to extend the content of our graph to other sources of information in collaboration with the Government of India to facilitate a larger variety of instances in Bharathi. Finally, our continued eforts on Bharathi knowledge graph aims to build a multi-lingual search across Indian languages to enable larger support and patronage for Bharathi.

References

[1] J. Hoxha , A. Brahaj , Open government data on the web: A semantic approach , in: 2011 International Conference on Emerging Intelligent Data and Web Technologies , IEEE, 2011 , pp. 107 - 113 .

[2] H. S. Al-Khalifa , A lightweight approach to semantify saudi open government data , in: 2013 16th International Conference on Network-Based Information Systems , IEEE, 2013 , pp. 594 - 596 .

[3] D. DiFranzo , A. Graves , J. S. Erickson , L. Ding , J. Michaelis , T. Lebo , E. Patton, G. T. Williams , X. Li , J. G. Zheng , et al., The web is my back-end: Creating mashups with linked open government data , in: Linking government data , Springer, 2011 , pp. 205 - 219 .

[4] L. Ding , T. Lebo , J. S. Erickson , D. Difranzo , G. T. Williams , X. Li , J. Michaelis , A. Graves , J. G. Zheng , Z. Shangguan , et al., Twc logd: A portal for linked open government data ecosystems , Journal of Web Semantics 9 ( 2011 ) 325 - 333 .

[5] L. Ding , D. DiFranzo , A. Graves , J. R. Michaelis , X. Li , D. L. McGuinness , J. A. Hendler , Twc data-gov corpus: incrementally generating linked government data from data. gov , in: Proceedings of the 19th international conference on World wide web , 2010 , pp. 1383 - 1386 .

[6] T. Lebo , G. T. Williams , Converting governmental datasets into linked data , in: Proceedings of the 6th International Conference on Semantic Systems , 2010 , pp. 1 - 3 .

[7] J. Höchtl , P. Reichstädter , Linked open data-a means for public sector information management , in: International Conference on Electronic Government and the Information Systems Perspective , Springer, 2011 , pp. 330 - 343 .

[8] B. Hyland , D. Wood , The joy of data-a cookbook for publishing linked government data on the web , in: Linking government data , Springer, 2011 , pp. 3 - 26 .

[9] R. Cyganiak , F. Maali , V. Peristeras , Self-service linked government data with dcat and gridworks , in: Proceedings of the 6th International Conference on Semantic Systems , 2010 , pp. 1 - 3 .

[10] P. Křemen , M. Nečasky`, Improving discoverability of open government data with rich metadata descriptions using semantic government vocabulary , Journal of Web Semantics 55 ( 2019 ) 1 - 20 .

[11] L. Bountouri , C. Papatheodorou , V. Soulikias , M. Stratis , Metadata interoperability in public sector information , Journal of Information Science 35 ( 2009 ) 204 - 231 .

[12] I. Petrou , M. Meimaris , G. Papastefanatos, Towards a methodology for publishing linked open statistical data , JeDEM-eJournal of eDemocracy and Open Government 6 ( 2014 ) 97 - 105 .

[13] H. Alani , D. Dupplaw , J. Sheridan , K. O'Hara , J. Darlington , N. Shadbolt , C. Tullo, Unlocking the potential of public sector information with semantic web technology , in: The semantic web , Springer, 2007 , pp. 708 - 721 .

[14] A. Subramanian , S. Srinivasa , Semantic Interpretation and Integration of Open Data Tables , Springer Singapore, Singapore, 2018 , pp. 217 - 233 . URL: https:// doi.org/10.1007/ 978 -981-13-2330-0_ 17 . doi: 10 .1007/ 978 -981-13-2330-0_ 17 .

[15] A. Subramanian , Semantic integration and knowledge representation of open data powered by linked open data , Ph.D. thesis, International Institute of Information Technology Bangalore , 2018 . URL: https://shodhganga. inflibnet.ac.in/handle/10603/262218.