The paper presents description of the problems related to a large amount of data frequently received from a Web-server by the Webinterface causing the insu cient performance of the latest one. A comparative analysis of methods for updating the data in the Web-interface was made, and the optimal method for updating the data in the realtime Web-application was chosen. The paper also provides an example of optimization of data processing using the data bu ering and implementation of this example in the modern JavaScript.
Multiple issues about insu cient performance exist in the modern Web-applications development. These issues include: slow rst page load because of large amounts of data; low responsiveness or even freezing of an interface when updating a lot of data, and others.
Web-applications can store more than 900 GB of data on the server side
(or in a database), for example, the data of the students from all around the
world [
This research shows the advantages of data updating implementation by using the Websocket comparing to other implementations of real-time data Webapplications and the e ciency of using an additional data handling optimization by application of bu ering.
Implementation of updating the visual data in a Web-client can be done by wellknown HTTP-requests, which are red by some user actions. For instance, this exact method is used in the search engine of the Google, USA: by pressing the search button, the user sends HTTP-request to the server, and after some short period of time, the Web-client receives the response and updates the information on the page. Listing 1 contains the example of a simple HTTP-request written in the JavaScript.
Listing 2-1. HTTP-request example, written in JavaScript import a x i o s from ' a x i o s ' ; //HTTP r e q u e s t s l i b r a r y import C l i e n t S t o r e from ' . / C l i e n t S t o r e ' ; // s t o r e o f Web c l i e n t / Function t h a t r e q u e s t s t he data from s e r v e r and r e c e i v e s an answer @ f u n c t i o n g e t I n f o F r o m S e r v e r @return f v o i d g / f u n c t i o n g e t I n f o F r o m S e r v e r ( ) f a x i o s . g e t ( ' / a p i / i n f o ' ) //Got ' p o s i t i v e ' r e s p o n s e with th e data from t he s e r v e r . then ( ( r e s p o n s e ) => f // Rewrite t h e data i n th e Web c l i e n t s t o r e
C l i e n t S t o r e . r e w r i t e C o n t e n t ( r e s p o n s e . data ) ; g ) //Got ' n e g a t i v e ' r e s p o n s e from th e s e r v e r ( an e r r o r ) . . c a t c h ( ( r e s p o n s e ) => f // Inform th e u s e r about th e e r r o r c o n s o l e . e r r o r ( ' E r r o r w h i l e r e c e i v i n g r e s p o n s e = ' ,
r e s p o n s e . data ) ; g g ) ;
Implementation of updating the visual data in a Web-client can be done by the help of HTTP-polling. In this case, the same simple HTTP data request is used, but it is repeated with a time interval, so that the Web-client can show the real-time information (Listing 2 has the example written in the JavaScript). This method is easy to implement and can be used when developing a simple Web-interface of a network device to observe relatively small amounts of data, for example, monitor one parameters table of a network device in real time.
Listing 2-2. HTTP-polling example, written in the JavaScript import a x i o s from ' a x i o s ' ; //HTTP r e q u e s t s l i b r a r y import C l i e n t S t o r e from ' . / C l i e n t S t o r e ' ; // s t o r e o f Web c l i e n t l e t r e f r e s h I d = n u l l ; // i d o f t he p o l l i n g i n t e r v a l / @ f u n c t i o n s e t I n f o P o l l i n g
@return f v o i d g / f u n c t i o n s e t I n f o P o l l i n g ( ) f r e f r e s h I d = s e t I n t e r v a l ( ( ) => f a x i o s . g e t ( ' / a p i / i n f o ' ) //Got ' p o s i t i v e ' r e s p o n s e // with t he data from th e s e r v e r . then ( ( r e s p o n s e ) => f // Rewrite th e data i n th e Web c l i e n t s t o r e C l i e n t S t o r e . r e w r i t e C o n t e n t ( r e s p o n s e . data ) ; g ) //Got ' n e g a t i v e ' r e s p o n s e from t h e s e r v e r . c a t c h ( ( r e s p o n s e ) => f // Inform th e u s e r about th e e r r o r c o n s o l e . e r r o r ( ' E r r o r w h i l e r e c e i v i n g r e s p o n s e = ' , r e s p o n s e . data ) ; / g ) ; g , 1000 ) ; g / @ f u n c t i o n s t o p I n f o P o l l i n g f u n c t i o n s t o p I n f o P o l l i n g ( ) f i f ( r e f r e s h I d ) f
c l e a r I n t e r v a l ( r e f r e s h I d ) ; g
g
Finally, the WebSocket can be an implementation of updating the visual data. In
this case, the Web-client establishes a connection with the server and subscribes
to some necessary topics (themes) of data, and the server sends the data when
and only when it is obligatory, for example, when the data has been changed in
the database. (Listing 3 contains WebSocket example implementation written in
the Javascript). Any Web-application with real-time data can be set as a good
WebSocket implementation (as in the short example), because the WebSocket is
the most performance-wise and server-load-wise e cient transport [
Listing 2-3. Example of the WebSocket session handling, written in JavaScript c o n s t DATA URL = ' data ' ; c o n s t GET SNAPSHOT RPC = ' getSnapshot ' ; / Handler that a b s t r a c t s the AutobahnJS (WAMP) methods @ c l a s s S e s s i o n H a n d l e r / c l a s s S e s s i o n H a n d l e r f / P r i v a t e f i e l d that s t o r e s s e s s i o n o b j e c t
@private / s e s s i o n = n u l l ; / @constructor @param f? S e s s i o n g [ s e s s i o n = n u l l ]
C l i e n t s e r v e r s e s s i o n . Null by d e f a u l t . / c o n s t r u c t o r ( s e s s i o n = n u l l ) f
t h i s . s e t S e s s i o n ( s e s s i o n ) ; g / Method that s t o r e s the s e s s i o n i n the
S e s s i o n H a n d l e r s i n s t a n c e @method s e t S e s s i o n @param f? S e s s i o n g [ s e s s i o n = n u l l ]
C l i e n t s e r v e r s e s s i o n . Null by d e f a u l t .
@return f void g / s e t S e s s i o n = ( s e s s i o n = n u l l ) => f
t h i s . s e s s i o n = s e s s i o n ; g ; / Callback f u n c t i o n that h an d le s s u b s c r i p t i o n messages @see http : / / autobahn . ws/ j s / r e f e r e n c e . html#s u b s c r i b e
Autobahn documentation @callback s u b s c r i p t i o n C a l l b a c k F n @param f Array g a r g s a r r a y with event payload @param f Object g kwargs o b j e c t with event payload @param f Object g d e t a i l s o b j e c t with event metadata @return f void g / / Method that s u b s c r i b e s to data updates messages @method subscribeToData @param f s u b s c r i p t i o n C a l l b a c k F n g c a l l b a c k F u n c t i o n f u n c t i o n that should be executed when r e c e i v e d a message @return f Promise g / subscribeToData = ( c a l l b a c k F u n c t i o n ) => f i f ( ! t h i s . s e s s i o n ) f
c o n s o l e . e r r o r ( ' Tried to s u b s c r i b e to d e v i c e data but no s e s s i o n was s p e c i f i e d i n SessionHandler ' ) ;
r e t u r n n u l l ; g r e t u r n t h i s . s e s s i o n . s u b s c r i b e (DATA URL, c a l l b a c k F u n c t i o n ) ; g ; / Method that r e q u e s t s snapshot ( f o r the c o l d s t a r t ) @method g e t S n a p s h o t @return f Promise g / g e t S n a p s h o t = ( ) => f i f ( ! t h i s . s e s s i o n ) f
c o n s o l e . e r r o r ( ' Tried g e t s n a p s h o t but no s e s s i o n was s p e c i f i e d i n S e s s i o n H a n d l e r ' ) ;
r e t u r n n u l l ; g r e t u r n t h i s . s e s s i o n . c a l l (GET SNAPSHOT RPC ) ; g ; g
The deduction (that most e cient option to handle big data is WebSocket)
appears when comparing the options described above using several criteria
(Table 1) because the size of the messages ow between the Web-client and the
server is the smallest and most justi ed at this point. The method of updating
data only by user action is not suitable for the application with real-time data,
and polling is not suitable for it when the data becomes bloated. Moreover, with
using polling, the Web-client always sends requests, even when the data in the
database was not changed. Multiple WebSocket protocol \wrappers" without
the standardization, on the other hand, are the apparent draw back for a
developer. In this case, the developer should choose a protocol supported both on the
Web-client and on the server side. Despite of the tiny amount of box solutions of
this protocol data transfer implementations, the WAMP (the Web Application
Messaging Protocol) [
To solve the issue of excessive messages ow from the server to the Web-client,
for example, in the case of monitoring a huge network, handling of each received
message should contain minimum amount of calculations. So that the overall
message handling time is decreased, which gives the Web-client more time to
execute other tasks (for example such heavy tasks as DOM (Document Object
Model) rendering) in the one-thread Web-client. This can be e ciently done with
using data bu ering. This kind of bu ering works alike the bu ering in a CPU[
Let us assume that Web-application (that is needed to be developed) can have either huge or small amount of data in the database. For example, in the case of network monitoring application with real-time data. If there is a low amount of network nodes (assuming their quantity is from 1 up to 50), the bu ering will not change much in terms of performance because the modern computers have enough calculation resources to handle telemetries of such nodes amount in a short time (if every message does not lead to recalculation and re-render of the network map graph, of course).
criteria Possible to cre- no ate an application with realtime data updates Big number yes of boilerplates (ready-made implementations) and, therefore, simplicity of development Web-client to server requests amount
Depends on the Requests are sent by an There is only one frequency of the interval, e.g. every 2 sec- request: to estabuser actions onds lish the connection to the server Necessity of Data ow can be The polling is used in The server sends implementation large only when the real-time data ap- the data to the of the big data there is a high plications; therefore, the Web-client by ow optimiza- amount of direct data should be updated himself; theretion methods in user requests. It quite often. On the other fore, if the server the Web-client should be handled hand, if the server con- observes freon the server, so tains too much data, then quent changes that other users the polling, even with of the data in do not experience the data clustering im- the database, all lags during work- plementation, might be- the changes will ing with the inter- come inappropriate deci- be sent to the face sion. Web-client as a large messages ow.
It follows that the bu ering in uence on visual Web-interface user experience is somehow needed to be minimized when there are not so many network nodes, or, more speci cally, messages from the server (when the messages come from the server less frequently than the bu ering interval). On the other hand, the bu ering impact on the frequency of the data updates is also needed to be increased in order to avoid stack over ow of the calculation operations and, thus, avoid the interface freezing when the messages start to come from the server more frequently. To implement this, the timeout of updating the data should be refreshed when no message was received in the bu ering interval at the shortest point and when the sum of the time intervals (including the time refresh-iterations) becomes somewhat critical to do a force update at the longest point. For instance, assuming minimum update interval is 200ms, if the Webclient receives only one message during these 200 ms, the data from this message will be applied to the data in the Web-client's store immediately after this time interval. Otherwise, if the Web-client gets more than 1 message per these 200 ms, the timeout will be refreshed and will wait for the next message again, then the timeout will be refreshed again etc. This will go on until the total sum of the time intervals becomes critical, e.g. equal to 1 second. In this case, all the data, which were merged from the messages (that were received during this 1 second) will be applied to the Web-client's data store. Thus, when the messages ow is large, the update occurs only 1 time per second and when the messages ow is smaller, the update can occur from as frequent as 1 time per 200 ms to as frequent as 1 time per 1 second.
Listing 4 contains the implementation of the algorithm above written in
JavaScript, EcmaScript 2015 standard [
i n t e r v a l = n u l l ; / i d o f t h e timeout with maximum i n t e r v a l
( when th e f o r c e update o c c u r s ) / m a x I n t e r v a l = n u l l ; / / @type f Boolean g
Boolean f l a g o f t he f o r c e update f o r c e A p p l y = f a l s e ;
Main data s t o r e i n s t a n c e s t o r e = n u l l ; / / / @type f S t r i n g g
Updating mode ( can be e i t h e r s e t to updating or accumulating ) mode = n u l l ; / / @type f Object g
Object with accumulated updates o f type f e v e n t s : Mapg / accumulatedUpdates ; bufferedUpdate ( updates ) f // s t o p p i n g ` i n t e r n a l ` timeout c l e a r I n t e r v a l ( t h i s . i n t e r v a l ) ; // adding updates ( accumulating ) t h i s . accumulateUpdates ( updates ) ; i f ( t h i s . f o r c e A p p l y === t r u e ) f / i f the f o r c e update f l a g i s s e t to t r u e
updating the data / t h i s . applyAccumulatedUpdatesIfNeeded ( ) ; // e x i t i n g the c u r r e n t method r e t u r n ; g t h i s . i n t e r v a l = s e t I n t e r v a l ( ( ) => f / c r e a t i n g ` i n t e r n a l ` timeout ,
a f t e r which the data update i s c a l l e d / t h i s . applyAccumulatedUpdatesIfNeeded ( ) ; g , Constants .DEFAULT UPDATE TIMEOUT) ; accumulateUpdates ( updates ) f i f ( ! updates ) f r e t u r n ; g updates . forEach ( update => f / e l s e , f o r each update add the data
to accumulated updates o b j e c t / t h i s . accumulatedUpdates . e v e n t s =
UpdateHandler . appendEventMessage ( t h i s . accumulatedUpdates . events , update ) ; g ) ; g s t a t i c appendEventMessage ( i n i t i a l V a l u e , newMessage ) f / Here , depending on the message s t r u c t u r e ,
new data about the network e v e n t s i s added / r e t u r n i n i t i a l V a l u e . s e t ( newMessage . id , newMessage . v a l ) ; g applyAccumulatedUpdates ( ) f c o n s t f e v e n t s g = t h i s . accumulatedUpdates ; // a p p l y i n g data to the s t o r e t h i s . s t o r e . d i s p a t c h ( e v e n t s A c t i o n s . updateEvents ( e v e n t s ) ) ; / i n i t i a l i z i n g the v a l u e f o r the accumulated updates / t h i s . accumulatedUpdates . e v e n t s = new Map ( ) ;
This algorithm was successfully implemented and used in the task of creating the Web-interfaces of network and geographical map that might contain up to 10 thousand of network devices and around 50 thousand wireless broadband links between them. The Web-server can send messages quicker than 1 message per 1 ms in the con guration with such a big network. Therefore the one-thread Web-client could not be able to handle and apply the data without a signi cant visual delay (when not using bu ering).
Applying adaptive methods of the data handling, on the other hand, allows one to increase performance while decreasing the calculations amounts for any width of the data ow. 4
Modern Web-programming technologies allow the developers to implement the Web-applications that do not have the excessive calculations and optimization operations. Especially, it is important when having a low amount of data but also do not have the signi cant calculations delay when having a big data ow. Di erent development tasks require using di erent techniques, methods, and approaches to handle information and optimize the data ow. Nevertheless, using the WebSocket and bu ering in the Web-client is an e cient way to organize and handle the data updating in the Web-application and to optimize the data ow.