The paper presents an approach to extracting the address and type of incident from a text data array formed based on posts in social networks. Data was uploaded from the Vkontakte community dedicated to incidents in St. Petersburg. A total of 48,943 records were collected and processed. A service has been developed for automatic recognition of the post topic and address extraction (if available) using natural language processing and machine learning methods using the free natasha library for Russian-language texts. Using the Geocoding API service from Google, the existing addresses were geocoded, and an array in GeoJSON format was obtained, which allows working with the dataset in various map services in real time.
Currently, the interconnection of heterogeneous urban elements is supported primarily by information technologies that provide communication links between residents, the management sector and infrastructure. On the one hand, the modern information space allows residents to constantly observe the life of the city, to meet their needs and interests in a mobile way, and on the other hand, it creates high expectations in relation to the urban environment and increases the level of responsibility of city authorities.
Due to the lack of capacity to limit the impact of hazards, many cities still face a high level of threats. As threats to cities increase, improving the resilience of cities becomes a major challenge. To increase the sustainability of cities, there is a growing need for information that is relevant to all stages of urban development. Thus, a better understanding of the spatio-temporal patterns of public response is a key step towards reducing damage and improving the resilience of cities.
So, a team of researchers from New York University analyzed incident data from two different
sources: from a traditional data provider that collects incident reports from multiple agencies, and user
messages from Twitter during Hurricane Sandy, which flooded many areas of New York in 2012 [
In recent years, interest in Twitter has been growing due to the fact that data in it is stored in real
time. Microblogs are increasingly attracting attention as an important source of information in
emergency management. Twitter is used as a way to predict accidents, natural disasters, and traffic.
[
The purpose of this work is to study and analyze the use of methods for extracting addresses from Russian-language text messages about incidents in social networks Vkontakte to generate geospatial data in the GeoJSON format, which can be used later in GIS systems or in the hardware-software complex “Safe City”.
The result of this approach is the distribution of incidents in the city on the map since the text will
consist of 2 entities: the address and the type of incident. At the first stage of the study, 5 main topics
were selected: Car theft, Accident, Fire, Robbery and Assault. Using this approach, we can identify
the most dangerous or problematic area in the city or find the area where the most theft occurs and so
on [
In addition, the data obtained can be used in the hardware-software complex “Safe City”, since often the incident message does not arrive in the system immediately, and this approach allows you to generate information online. For example, information about an accident can be included in the statistics in a week or even a month, if the accident was registered according to the Euro protocol. But active citizens quickly highlight road accidents and publish them into this community, which allows to search for problem areas in the city online.
In the case of St. Petersburg, we considered the possibility of extracting data about incidents that citizens write about in the social network called Vkontakte. The community about road accidents and emergencies in St. Petersburg (https://vk.com/spb_today) was chosen as a site for the study.
The collection of information for the study was carried out using a set of tools that included the Vkontakte API, a content parser, and a public service. The method API Vkontakte “Wall.get” returns a list of posts from the user's wall or community, using this method you can collect all the comments in the community. When conducting research on social networks, there is a complex problem of personal data security during parsing. Personal data, according to art. 3 of the federal law "On Personal Data", is called " any information related directly or indirectly to a certain or identifiable individual (subject of personal data)". No personal information was collected or stored in this report. A total of 48,943 records were collected.
For the collected sample, preliminary automatic processing of posts was carried out, which consisted in deleting entries starting with the words “News of our metropolis:”. In this case, we did not consider the daily news reports when forming the final sample for analysis, because its only summed up the results of the day. The size of the final sample after deleting such posts was 48,408 entries.
One of the tasks of the research was also the development of tools for automatic processing and analysis of the data array of posts from social networks. The main functionality of the tool was the automatic detection of the topic of the post and recognition of the address or its component part: street, block, house number, district, etc. To get the primary results of the tool, it was decided to recognize five topics of posts: Car theft, Accident, Fire, Robbery and Assault.
To develop the toolkit, we used the open neural network library natasha (v.1.4.0), which was
updated in 2020, for recognizing addresses in Russian-language text [
To recognize the topics of posts, we used self-written rules for the yargy parser. As a result of setting up the parser, a sample of these posts was obtained, which contained only five topics. The Figure 1 shows the ratio of posts with five selected topics to the total number of posts.
CAR THEFT
2% OTHER 69%
ACCIDENT 25%
FIRE 4%
ROBBERY
0% ASSAULT 0%
Based on this sample in semi-automatic mode (SQL query + manual markup), the average accuracy of the yargy parser recognition of the five selected topics was calculated, the results are presented below in Table 1.
The result of the average accuracy of determining the topics in the text using the configured yargy parser is good (more than 90% except for Robbery). Among the disadvantages of this approach, the long duration of the parser's operation time was highlighted, since which we can assume about the slowness of the algorithm itself, especially when increasing the data sample. Thus, the next stage will require optimization of the parsing rules and more efficient data processing.
To recognize addresses in the text, the built-in “AddrExtractor” function from the natasha library was used. Address recognition was performed on a sample of data, including posts with recognized topics. A total of 15,167 records were selected in the sample. To calculate the average recognition accuracy, the condition was set that if at least one part of the address is recognized in the text of the post (for example, street, name, house number, etc.), then the address is considered recognized. The results of address recognition are shown in Table 2.
The table shows that the result of the average accuracy of address recognition in the Russian text is satisfactory for solving the problem (more than 75%), but in the future, the algorithm and rules for address recognition also require improvements to improve the accuracy and quality of determining the recognized addresses. When recognizing the address, it was also noted that the highest percentage of accuracy is achieved when determining the name “Moscow” for the type “street”, for example, in the format “Moscow Street”. However, if you remove the marker word “street”, the recognition accuracy will significantly decrease, even if there are other markers nearby, such as “house number”, “building”, and other parts of the address.
As a result of the analysis, we can conclude that it is sufficiently possible to use open and free ready-made solutions that provide the functionality of flexible rule settings for performing tasks of this kind of analysis. 2.3.
Geocoding recognized addresses and formatting dataset to GeoJSON was the next step of work.
For this purpose, we used Geocoding API service from Google [
As we need to store and show topic on a map, we used the Feature and FeatureCollection objects according to GeoJSON specification. Here is the example of format we used:
{"features": [{"geometry": {"coordinates": [30.36091, 59.931058], "type": "Point"}, "properties": {"topic": "nan"}, "type": "Feature"}, {"geometry": {"coordinates": [30.516726, 59.73777], "type": "Point"}, "properties": {"topic": "Угон"}, "type": "Feature"}], "type": "FeatureCollection"}
The dataset of 15,1567 records with all recognized addresses was geocoded and converted to GeoJSON. Below in Figure 2 you can see the visualizations of points according to five incident topics. (e) Figure 2: Car theft (a); Accident (b); Assault (c); Robbery (d); Fire (e)
Meanwhile we discovered that not all the address parts were recognized properly, for instance there can be only house number without street location or only street, which can be long. Thus, for future steps of research work we need to increase recognition accuracy and quality of address parts extraction.
The selected source of information in the social network Vkontakte showed that users generate a
large amount of information about incidents. In Russia, Vkontakte is more popular than Twitter [
During the research work, a toolkit was developed for automatic recognition of the post topic and addresses. The considered experiment confirmed the good effectiveness of the selected open library natasha (v.1.4.0) with a yargy parser, which managed to extract the topic and address from the text of the posts. With the help of the Geocoding API from Google, we managed to get the coordinates of addresses, translate the result of geocoding into the standard GeoJSON format, which allows us to use this data in different map services, GIS, as well as in the hardware-software complex “Safe City”. In the future, it is planned to increase and improve the data sample by using methods of automatic collection and extraction of entities, improving the accuracy of extraction and recognition of the posts topics and address parts.
The work was done under the research topic of ITMO University No. 620179 “Development of a map service for monitoring the residents needs in the urban infrastructure development using automated data processing systems from social networks”.