Agent technologies currently play an increasingly important role in the information technology industry given its ability to learn and evolve, to solve information management problems, to employ data visualization and many other benefits. As a computer program, an agent deals with a challenge Internet users face every single day: to obtain reliable and effective data in the specific thematic field. Multiagent system consists of two or more autonomous agents and aimed at solving complex problems, such as Big Data, Data mining, primary structured and unstructured information processing (including text, numbers and multimedia types of data).
Science and technology is one of the most significant development directions within the scope of which technologies are created, solutions are implemented that effectively respond to the grand challenges. Problem solving in this area requires specific complex knowledge on a great number of research areas in conditions of incomplete information, limited resources and lack of time.
Big Data and Data mining class of problems describe knowledge acquisition problem from a numerous multidirectional information resources.
The emergence of information retrieval systems (IRS) is a natural reaction of mankind to solve the
Big Data problem [
IRS filling is conducted by agent technologies - technologies using special autonomous programs for solving various problems.
Nowadays in the development of intelligent information agent technologies, multi -agent technologies have been established to solve the problems of regular automatic collection and processing of structured and unstructured information in a certain thematic direction from various network sources.
There are three defining properties of Big Data – volume, variety and velocity. Referring to the amount of data, types of data and to the speed of data processing respectively. According to this 3Vs model, the challenges of big data management result from the expansion of all three properties, rather than just the volume alone - the sheer amount of data to be managed.
An agent system is a system created by an intelligent agent and is one of the Big Data proble m
solving [
There are invested additional properties in the modern definition of an intellectual agent: autonomy, sociality, reactivity and proactivity.
Autonomy refers to the ability of an agent to act purposefully to achieve a result without external control by systems. He has control over his actions and the state of internal variables.
The sociability of an agent lies in the interaction with other agents to accomplish a common task, and for consistency in resolving conflicts to reach a consensus.
Reactivity is a reaction to receiving an event from the external environment and adjusting its behavior according to the knowledge gained.
Proactivity means a desire to achieve the goal, improving the characteristics of the internal state nonstop.
Multiagent system (MAS) is the system of autonomous intelligent agents united in a common
network and their activity is also aimed at solving the tasks set by the system [
Due to its multitasking and structure, multi-agent technologies are used in many different areas: management of distributed and network enterprises, the educational process in systems of distance learning, complex and multifunctional logistics, etc.
A demonstrative example is the social network LinkedIn, which uses intelligent agent
technologies in recruiting to search for people based on their career information and research
activities.[
Projects in the field of mass online education, like Coursera, use agent technologies to inform users about events within a certain course.
When working in a global network, agents have to interact with diverse types of infor mation
sources [
Structured data: Data containing a defined data type, format, and structure (that is, transaction data, online analytical processing data cubes, traditional RDBMS, and even simple spreadsheets).
Semi-structured data: Textual data files with a discernible pattern that enabl es parsing (such as XML, data files that are self-describing and defined by an XML schema).
Quasi-structured data: Textual data with erratic data formats that can be formatted with effort, tools, and time (for instance, web clickstream data that may contai n inconsistencies in data values and formats).
Unstructured data has no inherent structure, may include text documents, PDFs, images, and video.
The degree of structuredness of the collected data is directly determined by the variety and intellectual abilities of the agents. When collecting a news article from an information portal, an agent can collect the whole source code along with ad units and other irrelevant information, and can highlight the components of a publication and collect data in its pure form, e.g.
The development of an agent for each information resource has some difficulties because of the lack of strictly regulated standards for constructing information resources and because of high flexibility of presenting information using html code. Therefore, different algorithms and standards are created for the placement of information for its correct identification: HTML 5.2, Selectors Level 3, ITS Version 2.0, etc.
Time passed by, and a problem arose in the use of agent technologies because the collection of data may occur against the desire of the internet resources owners, thereby the agent, without realizing it, could infringe copyrights.
In U.S. court practice; there have been cases where information portal owners have brought cases against collecting agents. For example, in case №17-cv-03301-EMC of United States District Court Northern District of California hiQ Labs inc. indicted LinkedIn for the fact that the social network has built protection against the collection of publicly available p rofile information in reference from United States Antitrust law. This case was initiated in response to the hiQ company's early accusation of illegitimate automated collection of LinkedIn profile information in reference from Computer Fraud and Abuse Act. Subsequently, a case judge Edward M. Chen obliged LinkedIn to remove the previously established protection on the collection of publicly available information by autonomous intelligent agents.
Afterwards the use of agents to collect information led to the practice of implementing the Robots Exclusion Standard (robots.txt, 1994). This is a file with a list of access restrictions for robots to content of an http server. The file should have a path relative to the site name /robots.txt. It reflects the availability of information to be collected by an agent.
To build multi-agent systems, it is necessary, among other things, to conduct a study of the
information environment[
At this stage, it is necessary to determine the available information about the object of interest and the legitimacy of its collection. It is also important to identify the available attributes of the object of interest in order to build the corresponding data model and relationships with other objects of the information source.
For example, in social networks where the person is the object of interest, communities, audio and video recordings can play the role of objects associated with it. In such cases, it would be necessary not only to collect information about objects of interest, but also to take into account the relationship with other objects, since in the future they can act as characteristics during intellectual processing.
The stage includes the study of mechanisms for placing information about objects of interest into an information source and the definition of an agent behavior algorithm for collecting information with maximum completeness. Any site is a set of interconnected web pages and its structure can be represented in the form of a hierarchical model tree, where the top of the tree is the ini tial page of the site and pages with the necessary information is the basis of the tree.
The purpose of the configuration stage is to list third-party software solutions to expand the functions of the agent. In particular, the Competitive Systems Analysis Department at MEPhI uses Python 3.7 as the programming language: on which the agent is implemented.
The essence of this method of collecting information is to obtain hypertext documents in html format using the http / https protocol and further identify the necessary data from the received documents. To develop such an agent modules are required, that allow you to access the information resource using the http / https protocol and a module for recognizing the contents of an html document. At this stage, the source of information considered fully explored.