The detection of changes in the web content and notifying users about them in timely manner is an important service for both the owners of the web sites and ordinary users. The system created automatically captures and monitors the differences between the content of the web page at a given moment of time and compares them with saved earlier. The system periodically parses the contents of files on a server, available for public access according to the given pattern. It compares its contents with previously stored version and in case if discrepancies are discovered, immediately notifies the user. Thus, the user will not have to monitor manually the specified Internet resources.
With every year, the information technology industry gains more and more rapid
development. Smart systems for monitoring, analyzing and managing things are being
introduced everywhere. However, the development of information technology is not
limited only to this area. All related industries are also developing at a rapid pace. In
order to increase productivity and improve the quality of the final product, the human
labor is replaced by automation [
The first systems that provided the tracking of changes in the documents were the
version control systems [
The work with SVN is not much different from the work with other centralized
version control systems. Users copy files from the central repository, works with
these copies and synchronize them with those that are left in the main repository,
commenting on new edits. The ability of two or more users working with the same
files is a useful feature. This is provided by creating local copies of the original files.
However, this product also has the function of blocking access to a particular file, so
that only one user could work with it. Also, the smart decision of this system is the
use of the mechanism of delta-compression – the saving of not the original and final
contents of the file, but only the differences between them [
The key features that SVN system provides for its users [
For a better understanding of what a version control system represents and what
possibilities it offers, see Fig. 1 [
Due to the strict observance of this plan the history of the project development is
created with the possibility at any time to return to a certain stage of development and go
to a different way [
SiteLock is protecting web pages in real time [
HTML files, a JavaScript resource and images. 3. Caching on the user’s side – keeping as much data as possible in the browsers of the site visitors for even faster display of content and offline viewing of web resources. 4. Loading pages from memory – pages with the largest number of requests are cached in RAM. This can be done without working with the file system and buffers.
Particularly useful for owners is the ability to self-control which data should be
cached, and which should not. So, in the case of site design changes or some other
global changes, SiteLock allows to delete all or only certain parts of the cached
content. Another strong side of this product is the availability of support. It works in 24/7
format, that enables users to get an answer to the question at any time. SiteLock
works well with its main aim – protecting users’ web resources and providing means
to improve their end products – that is why it became so popular. However, its
disadvantage is limited support of CMS systems [
The Content Downloader software is a parser. With its help it becomes possible to
save on computer the files with text and graphics presented on the site. In order to
work with it the knowledge of programming is not required, which is an advantage,
because it allows to include in the list of customers the ordinary users. This software
product, like its counterparts, is often used to quickly fill the new web page or to
download and compare different sites [
The software products considered above are not a novelty in the software market.
Each of them perfectly proved itself and became entrenched in its “realm”. Due to the
long time they have been in the market they have formed a clear range of services that
are needed for their users. After analyzing these services, as well as those which, due
to certain circumstances, are left without proper attention from the considered
systems, we were able to formulate requirements for automated monitoring system. In
the Table 1 the advantages and disadvantages of reviewed systems are described [
The aim of this work is to create intelligent automated monitoring system, so let’s first state the definition of “intelligent system” and “automated monitoring” notions. Intelligent system is a system of technical or programmatic nature that can independently solve its tasks from a certain subject area, creating a new knowledge based on the acquired earlier. The process of functioning of such a system usually results in constant decision making taking into account the current state of external and internal factors in order to achieve certain results. The functioning of such systems includes certain stages: 1. Clarification of the initial state at the time of task assignment to the system. 2. The comparison of this state for all factors and tasks with all previous states and tasks. As a result, there is a check on the possibility of finding a quick solution for the given task. 3. In the absence of the possibility to finding the response immediately, the system proceeds with decomposition of the task into a series of smaller ones, and repeats the actions described in paragraph 2, for each of these tasks. 4. The collection of the results from step 3 into one unit and the work results formation. 5. Saving the information about the task, the initial state of external and internal factors, the results of its realization to the base of the previous tasks.
The automated monitoring is a process of continuous collection of certain data (indicators, results of calculations, etc.), their analysis and the formation of further behavior depending on the results of the analysis. Also the key feature of this process is the complete absence of human interference. So, the intelligent automated monitoring system is a system that is created to further reject the need for a human participation in the process of gathering information, processing it and shaping the proper behavior in the particular subject area. The main use cases of automated monitoring system are shown on fig. 2. There are actors such as the User and the System. The User has two kinds of options: The receiving of notification as the message sent in a user-friendly format including the report about the monitoring results. Formation of the query. This option is somewhat wider and includes the more specific ways of interacting with the system. The case “Request Formulation” can be divided into two sub-cases. The user can either work with a user profile (email address, password, site name) or work with monitored parameters (creating new patterns and resources for tracking, changing existing validation rules, removing unnecessary functionality). The actor System has the following interaction options: Sending the report on the results of monitoring of relevant resources for the selected users. Monitoring case includes following sub-cases: ─ Parsing is the process of downloading the content from a web resource. ─ Comparison is the process of identifying differences with the content saved in the database earlier. ─ Report formation is presentation of differences (if there are any) in a userfriendly format. Work with the database case is recording, reading, editing the data stored there.
Activity diagram (Fig. 3) is used to display the sequence of all user actions with the system. The activity diagram is constructed in the form of an oriented graph. The actions are indicated by rounded rectangles. Decisions are marked by a diamond. If there is a set of actions that are executed in parallel, then the beginning and the end of parallel actions are indicated by a straight horizontal line. The black circle means the beginning of the process, and the circled black circle means the end. Activity diagrams are very similar to the flowcharts, so they can be considered as such. In the activity diagram it can be noted that the user of the system can perform a certain sequence of operations – registration, or logging-in, correction of available data in the system or the assignment of new ones. While in working in his personal cabinet, the user can see a list of all available web resources that he added to the monitoring list. It can also perform certain actions with them (such as deleting, editing, checking, pausing, etc.) or adding more resources to monitor. During the adding a new page, the user will be offered to select specific patterns of what is needed to track. He may also create his own. Those patterns represents static elements of the system (classes, their content and attributes). In the class diagram the package designation may also be additionally displayed. Also it allows to place some procedural elements, but this dynamics is better shown on other types of diagrams. A class diagram is intended to display the static structure of the system in terms used in object-oriented programming. It is best not to show anything on this diagram, except classes, interfaces, objects and their relationships. The class depicted in this diagram is determined by the attributes and methods of a particular set of instances. All instances of class are characterized by a common behavior and the same set of attributes (each instance of the class has its own set of values). Sometimes the name “class” can be replaced by “type”, but it is worth to note that these two names, in fact, characterize different things – the type is considered more general definition. Classes are depicted by rectangles signed on the top.
As a result of the process, the object can change its class or state. These two types of changes do not have radical differences from each other: objects belonging to any one class can firstly convert to a parent or branch class. In Fig. 4 the diagram of states of the object of the automated monitoring system is shown. Each state diagram has two distinct states of the object: the beginning and the end. The peculiarity of these two states is that it does not have any event that could force an object to change to its original state and does not have any such event that could return the object from the state of the end after it has been reached. Looking at the state diagram, it becomes clear that the monitoring system has two modes of operation: 1. The mode of execution of user queries – when the user is logged in and through the graphical interface interacts with the system. 2. Passive Query Mode – when the user does not specify directly what the monitoring system should do, it only receives a notification of the results of its work, according to the previously configured task.
The deployment diagram is intended to display the computing nodes of the system during its operation, as well as components of these nodes, which are also involved in the process.
The nodes of the deployment diagram are rectangular boxes with artifacts inside, depicted by rectangles. To represent a hierarchy of nodes, it is allowed to insert boxes. One node on such diagram can show a large number of physical nodes, as a cluster of all servers with databases (Fig. 5). The nodes are of two types: The device node is the physical computing node. It has its own memory and other means to run the software; The runtime environment node is a virtual resource that runs on an external basis and is designed to execute other software elements. The deployment diagram for the automated monitoring system contains four nodes: The user’s browser through which interaction occurs. The system itself (its hardware and software). The resource server which provides monitored data. Database server.
Since the system is should preserve the results of monitoring and compare them with previous results, the system requires a large amount of memory. On the early stages of monitoring system usage it would be enough to use one server to execute software queries, but with the load growing, the servers will be distributed. Also, the advantage of this solution will be to spread the database at different nodes to ensure reliability and stability of the work. 4
The working of monitoring system is based on the server-side utility for UNIX-like operating systems cron. It serves to execute scripts with a certain periodicity or for one-time execution at some specified moment. Under its command, the script cron_check.php is performed, located in the root file of the project, which in its turn goes through all the resources added by the users and determines whether it is time to re-verify them. If the time comes, the url adress of the resource and the regular expression by which the data on the page should be found is taken. The obtained data is then compared with those stored in the database, and in the case of detecting changes the user who added the resource for verification is informed.
To enable automated monitoring system to fulfill its purpose it needs to set several initial parameters:
All these items can be specified on the Add page. The main element of the user interface of the monitoring system is a table with a list of the web resources added to it, general information about them and a set of functions for the work (Fig. 6). Since each page in the global network has its own domain there is an additional grouping of custom links behind it to visually separate the various resources. On this page, the user has the following set of functions:
(delete) removes links from the table if resources with such domains do not exist, removes and deletes also all data from the database about it;
(scan) performs an immediate check of a given resource based on the specified regular expression (the result of this check is not stored in the database). If changes were detected, the system will provide links by which these changes could be reviewed (Fig. 7);
(update) performs parsing by pattern in order to create the basis for future comparisons. It will be especially useful in case of finding the differences that are permissible for the user and he would like to put them into the system; (change status) contains one of the two icons. The user switches them depending on whether he wants the resource to automatically pop up or not. If the user decides to activate the link, a page similar to that in Fig. 8 will open. The differences between two texts are highlighted The page headline gives a brief guide to the correct interpretation of what is shown below. Also in the top is the list of hotkeys for quick navigation is shown.
In case of acceptance of changes, the user updates the record by clicking on the correspondent button on the resource page. In the case if the detected changes are not caused by user actions or are not approved, the user proceeds to removing them.
It is also possible that the changes occurred, the system found them and informed the user, but due to certain circumstances, he failed to react to them. In this case it would be helpful to view the history of all changes on the register page (Fig. 9). Also, this page provides an additional functionality:
(compare with the standard) is the comparison to the state of the resource at the moment of its adding to the system.
(view recent changes) shows the changes that occurred on the site at the time and those that were detected during the pre-check.
The intelligent system of automated monitoring of changes in web resources is called the intellectual exactly because of the ability to analyze user behavior and based on the collected data to predict its future actions and partially perform them instead of the user (of course, with his prior agreement). In this system the decision-making mechanism is assigned to an artificial neural network. The Fast Artificial Neural Network Library (FANN), a free neural network library with an open code that realize multilayer artificial neural networks on C was taken to perform this operation. This resource provides artificial neural networks more than on 20 programming languages. Below is a list of features of this library, being the reason they were chosen for development: the ability to develop a topological training that dynamically builds and trains an artificial neural network; easiness in use – gives the ability to create, train and run a network using three functional calls); high speed performance; possibility of setting on the fly with a lot of parameters; well documented; realized several different activation functions; open source, but can still be used in commercial applications (licensed by LGPL); language binding for a large number of different programming languages; widely used (approximately 100 downloads per day).
The programming language which is used for the interaction with this library and for the monitoring system is PHP. The neural network learns to recognize the following types of differences: critical – in case of detection which the user must be immediately notified; data about it is kept (required for tracking); interesting – the system notifies the user, but don’t add information about it in the register of changes (their monitoring may be disabled at the request of the user); not interesting – is ignored
It should be noted that any differences which the system will find for the first time will be marked as critical. When viewed, the user assigns one of the proposed statuses. Further, when neural network has learned to the satisfactory level, it will be able to independently review the differences and determine whether or not to notify the owner of this resource. In the early stages of the network, each distinction will necessarily need to be approved for some status several times, to ensure correct weight adjustment. 5
To ensure that all components of the system function properly, will tested it on variety of web resources, making changes and verifying whether the system can detect them. As an example we use the main page of the site sudentinfo.net, and a pattern for monitoring its entire contents. The first step is to add this web resource into the system (Fig. 11). Next, go to the My records page to confirm the success of creating the reference resource for the resource added the previous step. In Fig. 12 it can be seen, that the resource will be monitored by the system every hour. Next, we checked the newly added page for data dynamics – whether there’s a plug-in or extension that constantly changes its content, such as a clock or a visitor count. To do this, we clicked on the “magnifying glass” in the line of this entry and saw the result in Fig. 13. Now let’s fire the index.php file, which is responsible for the main page of studentinfo.net and add the text “sitechecker.sc.ua” to it. Put this text in the <head> tag – it hides it in the user’s browser, but in the case of monitoring, the system should detect these changes. Then again we will test in manual mode, so that we do not wait for an hour. The system detected changes (Fig. 14) and organized a link for their viewing. By clicking on the link the user can see that the system correctly recognized the changes made and showed them as it was intended (Fig. 15). So the system worked correctly at all stages of the test. It certainly can’t be considered a hundred percent guarantee of its work, because in real conditions it will have to do dozens or even hundreds of more complicated inspections per second, but according to the results of testing we can say that the development of the system goes the right way. After analyzing three different software products (different in structure and specialization, but similar in the functions), which have long been on the service market and are well known, we summed up their strengths and weaknesses and highlighted the functionalities necessary for created system. So: Subvertion is a great means for finding differences in files and demonstrating them to the end user. It is also worth noting the mechanism for storing and transmitting these differences when not all file contents are transmitted. This saves resources; Content Downloader is a multifunctional parsing software that has a large set of custom settings for downloading both entire pages and their small elements. These patterns will be reused in developed monitoring system; SiteLock is an application with a large set of different scanning algorithms that allows it to track even the smallest changes and make decisions independently depending on the type of changes.
As the result of this work the practical implementation of software system monitoring the changes in web content was created. We used following software platforms for system. The software was written in PHP; HTML and CSS; JavaScript. It was executed using Apache HTTP Server; MySQL; Debian – the operating system on the server under which the web server will be deployed; Webmin – a tool for managing the operating system through a web interface. This paper contains detailed description of initial analysis, system’s functions and created software product. An example of changes monitoring system operation is also provided.