In the work an implementation of the information system of content monitoring of web resources is presented. Means and methods on the basis of which this system is implemented are analyzed in details. The structure of the created database - types of data, connections and restrictions was developed. An algorithm for data collecting from external web resources based on user's musical preferences was developed. As a result of the practical part of the work, the information system was developed in a form of a web application. A description of the software components, its functional purpose, logical structure, used technical means, call and download techniques, appropriate input and output data formats of the system are given. The functional implementation of web application security is being considered. The user's manual for the developed system is described in details. The main functions of the system were tested and the results of the analysis are presented in the form of a control example, which demonstrates step by step the work of the developed system.
The Internet is a system that combines smaller local area networks and individual
computers into one large global network and provides communication between them
through a variety of network protocols [
There are sites that are not distinguished from other sites with the same subject
matter, the contents of which are almost identical and often irrelevant. In this regard,
the search of high-quality material is significantly complicated for the right user [
Due to Internet popularity increasing and correspondent increasing of number of
web resources with diverse content, the process of updates tracking with aim to find
necessary information complicates over time [
Content is a set of information that exists only in electronic form and is indivisible in
time. Content monitoring is a set of operations on the Internet information which
includes search, analysis, classification and clustering of data [
Today, there are many information resources whose purpose is to provide information on events in the world of music, such as concerts, festivals and others. Among the users of Ukraine, the most popular are such web-services as Kasa.in.ua, Concert.ua, Karabas.com. Web service Concert.ua (www.concert.ua/ua) focuses on the sale of tickets for various musical events. Here you can find various filters to find information based on many criteria, such as location, price, event category, and musical style. This is what distinguishes this web service from its rivals. Events on this resource are added exclusively by moderators. Also, it should be noted that this service specializes only in music events and here it is not possible to find information of another category. This service has quite large developed communities in various social networks and a large amount of advertising to popularize the service. Among the advantages of this web service, you can highlight the following: archive of events; search field; narrow specialization; developed customer support service; developed communities in social networks; large number of filters developed communities in social networks. Among the disadvantages are the following: content is rarely updated; adding new events exclusively by moderators of the site; difficult navigation; there is no e-mail distribution. Also among the popular analogs you can select the web-service Kasa.in.ua (http://kasa.in.ua/). This service has a wider specialization. Here you can buy tickets not only for concerts and festivals, but also for theatrical performances and various sports events. New entries can be created by both resource moderators and event organizers. On the site you can find the search and filtering of events by city, which will allow to some extent to accelerate the search for the desired event.
Among the advantages of this web service, you can highlight the following: adding new records as moderators and event organizers; the presence of a search field; stable content upgrade; convenient navigation; developed customer support service. Among the disadvantages are the following: wide specialization; filtering events only outside the city; lack of mail distribution; lack of records archive. An even wider range of event categories includes the Karabas.com web service (http://karabas.com.ua/). Here you can find information about festivals, concerts, seminars, theatrical performances, sports events, tours and even quests. You can find a large variety of filters and a search box for easy search on the site. Since the specialization of the web service is considerably broad, the update is not so frequent. Among the advantages of this web service, you can highlight the following: developed customer support service; search field; large number of filters; developed communities in social networks. Among the disadvantages are the following: there is no archive of events; difficult navigation; adding new entries to site moderators only; wide specialization; content is not stable; There is no e-mail distribution. For a more detailed analysis and comparison of the advantages and disadvantages of each of the described web services, I build a comparison table with the list of features required in web systems of this type. As a result, it is possible to distinguish the basic functions which will be obligatory when constructing a system of content monitoring of web resources: convenient navigation; the possibility of adding new events by event organizers; frequent updating of the content; availability of events archive; availability of a search field; availability of filters; E-mail distribution. 4
Goals tree for the projected system is shown in fig. 1. The main purpose of the designed system is a content monitoring of web resources. The main goal can be broken down into the following sub-targets: the first sub-goal of the system is to save URLaddresses of websites. The system administrator makes URL-addresses in the database. The second sub-goal is to get user data. In turn, it can be divided into the following steps - registration or logging in to the system and entering data about the musical tastes of the user. These steps are taken separately, since the selected musical tastes may eventually be changed by the user in the system settings. The third subgoal of the projected system is the gathering of information from the websites. This subcategory is accordingly divided into the next steps - the transition to a website and the creation of a query to search for the musical tastes of the user. The information is collected from sites whose URL-addresses are stored in the database. The generated request is sent to the relevant website and the necessary information is searched. The fourth sub-goal is to store the collected and generated information in the database. The fifth and final sub-goal of the projected system is the sending of final information to the user. This sub-target contains two steps:
The formation of the final information for sending to the user - the system extracts the information required by the user from the database; Sending final information to the user - data extracted from the database, sent to the user's page, as well as in the form of a letter to the e-mail.
We can conclude that the gesigned system should directly affect only on the database.
This connection is important, since well-formed and stored data collected will be
much easier to choose and submit to the user. The purpose of the projected system is
easier to understand if the purpose of the system is correctly formulated. One can
formulate the following goal of the projected system: the system must collect data
from the sources specified by the administrator, generate and store the collected data
in the database, if necessary, to send data from the database to the user's system page,
in relation to its tastes and desires. To simplify the design and construction of the
system, it is necessary to decompose it and highlight its main components. [
The next step in system analysis is to construct a diagram of activity, which will depict the main activities of the system and will describe in details the process of the system (fig. 2). On the constructed diagram of the activity is shown 5 basic processes of the system. The process of filling out musical tastes data involves entering data by the user at the registration or later on the settings page. The process of viewing information about a particular event involves extracting information from the database and forming a response on the user's page. The process of viewing information about all events involves extracting all information from the database associated with the preferences of the user and forming a response on the user's page. The manual information collection process involves the collection of information from websites and is important in cases where the collection of information did not occur automatically. Creating a new event involves entering user event data and saving it to a database. The state diagram depicts a change in the states of the system when performing its processes (fig. 3). According to the projected system, if the information about a particular event is already in the database, it immediately returns to the user and the system goes into the final state, which can still be regarded as a state of expectation. If the information collection process is carried out, then the system goes from state to state according to the stages of the process of gathering information. As a result, the collected information returns to the user's page, and the system enters the final state. The following diagram - a sequence diagram (fig 4) - displays the sequence of sending requests in the system. This diagram allows you to better understand the process of creating a system. From this diagram we see that the information collection service exists independently and is used only when it is necessary to collect information. When registering, the user enters into the system data about his or her musical tastes (message 1). The system, having received the necessary data from the user, creates and sends a request for the collection of the information collection service (message 2) and receives the information collected in response (message 3). In turn, the system returns the received data to the user (message 4). Also, the user can send the system ID of an existing event system (message 5) and immediately receive information about the given event (message 6). To view the projected system from the point of view of software and hardware implementation, a class diagram (Fig. 5) and a deployment diagram were constructed (Fig. 6). According to the class diagram shown, each event has its own category, and each category contains many events. Also, the diagram depicts the classes responsible for handling events: the Class EventRepository contains the basic methods for working with the database, or more precisely for working with the event table of this database; the EventService class contains business logic for creating, deleting, and presenting an event; the EventController class is responsible for receiving requests from the user and sending a response.
The main purpose of the system development is the creation of a web application that can collect information from other web resources, compile and record the collected information into a database, and most importantly - send the user records that will satisfy his musical tastes. The system being developed is a multi-user system. There are two types of users in the system: the administrator and the user. A simple user has the ability to view information about a particular event or all events generated and sent by the system, as well as fill in information about their musical tastes. The administrator has an advanced set of features, as it can, in addition to the functions listed above, also manually create records and run the process of collecting information from external web resources manually. Table 2 illustrates a list of user types and functions available to users. The ability to create new records is only allowed by the administrator to ensure the reliability and reliability of the information provided by the system. The ability to start collecting content from external web resources manually is provided to provide information updates in the event that the process does not start automatically according to the established schedule. The developed system can be used as a free webbased service to advise users of events in accordance with their musical tastes. This web service can also be expanded to include paid revenue generating features for owners. This web service will mostly be used for entertainment services. This problem is well-known and widespread in the modern world, since it becomes more difficult to find the necessary information on the Internet every time. In this case, the problem of finding content about concerts and other events in the world of music is being considered. The problem is important, as the user will have to spend a lot of time and effort to find the right content for himself and make sure that it is reliable. 5
Before developing a system, it is necessary to consider and select a method for content monitoring and a database structure to store the information collected.
The parsing process consists of three main steps:
Collect information in its original form. Automatic parsing means copying the
code of a particular page with the subsequent extraction of content from it
(necessary data);
The stage of extraction, further processing of information and changes in its
format. Processing, systematization, categorization of the collected information and
bringing it into one kind is carried out [
In order to collect all the necessary information from a website, you need to
bypass all its pages and highlight the necessary content. Bypass all the necessary
pages of the site can be done in different ways:
By processing each page, the parser can be trained not only to extract the
necessary data, but also to temporarily store all the internal links that occur in it on the
path. Next, the parser goes to the pages of the website for each of the links
collected and extracts data from them until all pages are gone; [
Regular expressions are used only for the extraction of data that has a strict format
- electronic addresses, phone numbers, very rarely - addresses, template data. [
Taking into account all the advantages and disadvantages of the above methods of data extraction, it was decided to use the Jsoup library to bypass the DOM tree and develop a separate parser for each of the websites.
All data collected from websites will be generated in Event objects and translated using ORM into a database entry. In the same way, the records read from the database for presentation to the user will be converted to objects of the Event type. A relational database will be used to store the collected data, which will provide a rigorous structure of stored data, which is necessary for a single representation of all events in the system. Also, relational databases can support more users than non-relational ones. Although the use of a non-relativistic database will allow the storage of more unstructured data that does not match a single data format, and the use of a relational database will require greater performance if you store a large amount of data. Relational data model displays data in the most comprehensible form for the user and is based on a mathematical apparatus, which allows a succinct description of the basic operations of the data. After analyzing the variety of tools for solving the task, the system of content monitoring of web resources is resolved to be implemented as a web application. The process of developing such systems consists of two main stages (Fig. 10): creation of the user interface (client part); creation of the server part. The development of the client part involves creating a convenient interface for connecting the user with the server part. It is assumed that the user works only with the interface and does not see what is happening on the server. Everything that the user sees - forms, windows, colors, sliders, fonts, animations - belongs to the client part. The main tools for creating a user interface are tools such as HTML and CSS. To develop the server part of the web application, you need to choose a programming language that will satisfy the developer's requirements. Among the popular server languages today, Ruby, Node.js and Java can be identified. The main frameworks for developing high-quality web applications in Java are Spring, Play, Dropwizard, Spark. 6
The practical result of the work is a developed and ready-to-use web application for monitoring web content. Namely the intellectual information system, which is aimed at gathering information about concerts, festivals and other musical events on the territory of Ukraine. The server part of this web service is written in the Java programming language version 8 using the Spring framework, including Spring Boot, Spring Data, and Spring Security. The client part, that is, the user interface, is developed on the basis of the Bootstrap framework. The developed system is deployed as a separate isolated docker container, which ensures not only the ease of deployment of the system, but also reliability and security. A relational database with a set of data is necessary for the correct operation of the developed system. That is why the MySQL database server is deployed in a separate container and linked to the system developed. As the system deploys as a separate docker container, the pre-kernel file should have installed the JRE container, that is, the Java Runtime Environment that is freely accessible. The purpose of the developed system is to collect the information necessary for the user from various web resources, its further processing, grouping and returning to the user. The web-service focuses primarily on collecting relevant, reliable information about festivals, concerts and other musical events on the territory of Ukraine. Further, the information collected is grouped by categories and musical styles and sent to the user according to the musical tastes introduced by him during registration. The developed system can be expanded and improved by introducing a new functional.
During the development of the system, the following technologies were applied:
Spring Boot - Allows you to quickly configure all major project dependencies and is the main mechanism for developing high-quality web services; Spring Data - provides access to the relational database, simplifies the connection setup and provides a ready-made feature to perform basic operations; Spring Security - Provides the ability to secure the application, build authorization and authentication systems; Spring Retry - Provides re-execution of part of the code in the case of certain foreseeable exceptions (for example, in case of an error connecting to an external web resource); JSP - technology for generating HTML pages with dynamic elements associated with the controller; Apache Tiles - a technology that allows you to generate static blocks of HTML pages and only download them when first launched (such blocks are usually header and footer); AJAX - technology for building a user interface, which allows you to send requests to the server and update the page received in response to data in the background without reloading the entire page; Pattern of designing DAO (data access object) - makes the logic of interaction of the system with the database to a separate level, thereby separating it from business logic. Data operations are implemented without disclosing the details of the database; design pattern DTO (data transfer object) - allows you to transfer from the level of interaction of the system with the database to the level of business logic, only the necessary part of the extracted data, which is less resource consuming; jsoap library is a Java library for working with HTML. It provides a very convenient API for extracting and processing data using DOM, CSS and jQuery methods.
In the process of constructing a relational database structure for the developed system, the following entities were identified:
EVENT (id, name, date, price, image, city_id, category_id, website_id); CATEGORY (id, name); CITY(id, name); USER(id, name, surname, email, password, role, enabled, uuid, category_id); WEBSITE(id, name, url).
The CATEGORY and EVENT essences are linked by one to one (1: M). The essence of CITY and EVENT is one-to-many (1: M) with each other. The essences of EVET and USER are linked to each other by the link many-to-many (M: M). The CATEGORY and USER nature of the relationship between many and many (M: M). The essence of WEBSITE and EVENT are linked by one to one (1: M).
The underlined attributes of the above entities are the primary keys. For primary keys, the value of NULL is inadmissible. Deletion is limited, but cascading recovery. The developed web application is a distributed client-server system. The list of system functions is listed below.
Fig. 9. A filters set on the event viewer page 5. Create new event records, edit existing records and delete records from the system administrator (Figure 10). 6. Blocking and deleting admin users (Figure 11). 7. Search for events by name, category, city of holding and ticket price (Figure 9). Fig. 11. User Access Control Window
9. Adding new event records from other websites manually by the administrator. It is known that there is always a risk of cyber attacks on the website. This, of course, can negatively affect the security of user data and the reputation of the website owner. Therefore, during the development of the system was the most common methods of attack. SQL injection is one of the popular methods of hacking sites through the introduction of arbitrary SQL code in the query to the database. The system is protected against this type of attack due to the use of parameterized queries and sending parameters to the server separately from the body of the request. CSRF Attack - Interspace Forgery Request - A type of attack on website users that uses the shortcomings of the HTTP protocol. To protect the system from this type of attack, arbitrary CSRF tokens were used, which are verified by the server. To prevent errors when creating new entries in the user database, all fields are validated. Also, when registering, the user must create a password that contains at least eight characters. A rooted web application is cross-browser and cross-platform so it can be used on any device that has a browser installed and has access to the Internet. The input is the user's data that he makes during registration - his name, surname, email address, and a list of the genres of music he prefers. Output data, in the form of records of recommended concerts and music festivals, is based on the input list of the music genres that the user likes and displayed on his page.
The purpose of the system is to collect information from external web resources with further processing, grouping, storing in the database and sending to the user's page. In order to check the efficiency and correctness of the work of the developed system, as well as the compliance of the declared functional of the reality system, a control test was made consisting of three steps described below.
Step 1: View one of the websites, with information about the concerts, festivals and other musical events (fig. 12). This work opens the basic concepts of this topic, such as content and content monitoring, are described. The main methods of content monitoring are presented, which include specialized programs or parsers, news subscriptions, use of search engines and manual data acquisition. The advantages and disadvantages of using these methods are determined. As a result of the analysis it is determined that the most effective method of content monitoring is the use of specialized programs (parsers). The main stages of the implementation of parsing web resources and the purposes of their use are listed. The system is a web application with the following functionality: account creation; entering information about musical tastes; gathering information from other web resources about musical events, festivals, concerts; create new events by yourself; display of the proposed musical events system on the user's page. The system is proposed to be used in everyday life. The system can be used by people regardless of age, gender, musical preferences and will be useful for all concert and festival lovers. The developed system will be freely accessible, each user will be able to use it completely for free. But since for the development and updating of the system and the payment for its maintenance costs are necessary, it is possible to add to the system of paid functions or collection of donations for system development, attracting investors. Therefore, I believe that the development of such a system in terms of relevance and payback is appropriate. This system is expedient in terms of acceptance by end-users as it will considerably reduce the time of search for the information necessary for the user about the cultural events that will be held on the territory of Ukraine.