=Paper=
{{Paper
|id=Vol-1536/paper17
|storemode=property
|title=
Подходы к повышению пертинентности информационного предложения в медиасервисах на основе обработки больших объемов данных
(Approaches to Improve the Pertinence of Information in the Media Services on the Basis of BigData Processing)

|pdfUrl=https://ceur-ws.org/Vol-1536/paper17.pdf
|volume=Vol-1536
|dblpUrl=https://dblp.org/rec/conf/rcdl/PhilippovZSK15
}}
==
Подходы к повышению пертинентности информационного предложения в медиасервисах на основе обработки больших объемов данных
(Approaches to Improve the Pertinence of Information in the Media Services on the Basis of BigData Processing)
==
<pdf width="1500px">https://ceur-ws.org/Vol-1536/paper17.pdf</pdf>
<pre>
                      
               
              
                                                                  
                                      
                           !"#
                                        $  
stanislav@philippov.ru            VZakharov@ipiran.ru                                         ssa@ipi.ac.ru
                                 dm.kovalev@gmail.com
                                                     BC + 3  DEFGHIEJ K -+#
   % &'#  %&  &(    %              ,+4      K                2&(
 ) & %   %   # *  +,          .  -  2  2 &  
  & ''        (,-           . (          '        +1  '#
 . /   0    &             1     % #           )  0
 1) 2  %     3 4  %          . '  + (9  
# %&   & +1'  + 1        .  %  1   ) % '# 0 
       '   (  (           , 5*          1 2  . 
-%1(#  . +( #  (         6 5*     ) 2 ..
    ( (     &     +1-          5  2      )+-    ,'      1&)
 & +( (#   (       1 '-            (,'       ' ''  1&
& & +                     5       & + 2     (#       ) -62
   & +'       +1  + 2            .          &1      5 
 & )'  '    &(             3 #      %4#   (  & 
    +1-'  . +(  % (  (        '   &    & '-
  ' &(#  ) -6 (               . + & '       +1  '        *      '
   1            ,                 & +( ( 6   1 
  1 & +  1 )             + %         '&  1 &2 '2   
 (     ' +, %  )  .27              +1  '&     (
  (    & %(               %      +1  ' B&1' 9
 %  . (  1 &'8 1( 2#               ''        ('            +1  +
 &     0  ( 1 '-           & )2#   (#           -        )&+#
 ,%+ )1  9 ,'  . #         ( -'  1 +  1        &'#
 #  %  )+ 2 - -  -             %     +1  '#   . ( % 
&91 (   1.#   (  &2  +             +1  2    9 & )' 
    '' & 2  (  (         (    %
   & &(           (' -        +1  +
 & )2#9 &   ( . (            !"#                 
    '  & + 2 ( 1  %            
       %  &  :;<; " 
 (                 &&9   $              '6     '   & +(
 1 '              "#    +(2        1-'  &-6       (
&   =>?@>A60414X0139.                    & &(    & '     (' '     +1  +
                                                          & )2LM#NO8
          
                                                         x    1 (  &.
   ", 1&)      (,'                3Non-
 & 92  % &' & 1 +(               Personalized Recommenders);
   (   2   0   2
                                                         x  ' +.'3PJIGEIG
 .#        1 ' 6  
                                                         >H<GEFHIQ4R
  ) +1 1(&% )
                                                         x    '         +.'    & 
$%  XVII & %  '                          1 & )2 %         +1  2  
DAMDID/RCDL(!)* +   %                      9  3STEF-User
                                  Collaborative Filtering);
 #     ,- .  -    13-16               x     +1  ) 
!)*                                               . 
                                                           (Metrics and evaluation);


                                                   114
x    '              +.'          &             (+      (              +1  -
    1 1  '12 9& 5 (Item-                         & +(         (       ( -
   Item Collaborative Filtering);                                   1) +( 5( 1  &( &(. B
x  &.                            %               0   ( &2 '           +1  2  &' 
    1. . (Matrix factorization                          .  & )2. B              '
   recommendation algorithms).                                      &(         +1  + 2         (,'
    +,2             ' +-   1.                      + & 92                  %,
 & +(                                  (      (  & + 2  2.
 .  % &',2 &+             +1' & &                        )            9       
   '  +.'                 1 )( %                & +(2     (imhonet.ru),
 .'      (User-User       CF,    Item-Item       CF).           %         )    %         1 '
          & 92                     %             ( +  + (') 1             &
 9&'       +1  2 (..  +)                    1 9'  1 & +  [3].
    +1  & %             & &  ) '                    7            6(   +,
 (  2  &.2     1                          &% &          &  ,'        
   &) %   ',                 +1  2,                (,'    .   
          +1  2          &-6           9               +, 5 &(   1+
 / & )'.              & %  &                     '(2        2            Hulu
 9 &         9'         , )          +1  -            (http://www.hulu.com/) & ' &      
& (  (,  &-6  &(                           1 ) %       &   (  1 ( , ,
  , )  9   (                         (,  +(  . .),   (2 ( 
  (   ')   (,   (                               400               1             )
   +1-'         ' +-             +1  2                 &      2  &             
 9 & )'.                                                (  .
   79      &.2  %
)( +'      (       1(     &       +(            3 .      
 (,       -)'  )   -6
       ( , ( & '    ),
                                                                            '  
 6-         ' +                     2,              Hulu
 &.  1 & 2  %)(  
                                                                         '   '  +,   &(
  %  &% . 7,  ,  
                                                                    Hulu          +1'        Apache HBase
[&.$1(         (https://music.yandex.ru/)        
                                                                    (http://hbase.apache.org/)                ) +
    &.2 )(                    1.
                                                                    1( '                  Hadoop.
1( +(             ),             1
                                                                     %1. &( HBase  %)
     2,   ( (               +1  -,
                                                                     ,  0 &           Google BigTable, 1 ''
 &. &12           +1  ' (1  . +(
                                                                    0   +   .,  &96
2) [7]. 7 1 ,                     
                                                                     &(      (')  .
   , 2             1(               +1  
                                                                    (    '         .' &    &().
    9( 1(  
                                                                    /6                 &        -),
     2            & ). ]           1 '
                                                                      (2 & '   2 2 (2  .
 .'             +1  + & )'.
                                                                    )   -)  %          +1 +'    
 '  % ) ( & +   +- . +
                                                                    ) ,   1 (  9( (
 & )2              +1-'      &       +(
                                                                    &(. ( ''              ,   (
&(,  ,  , . « '» 
                                                                     5&'-' 2   & (  
«      '»,         (       9          +
                                                                    (column families). B  1&  .( & 9(
1( +( ,  + ,                 ' 
                                                                    (+      & (  )(2  -)  2 
. ( 1( +( 9. B                    .  
                                                                     . .  .( 2   % &  '+'
   , 2,  & +'   
                                                                         &  , ) &   f  
)(  1 )( &2 '              +1  ' 
                                                                    % 2.     Hulu !f HBase
    +1   .  ,                   
                                                                         +1'         (         
1( +(            (  + ,        &   
                                                                    Facebook (www.facebook.com), Twitter (twitter.com) 
& )  &  '                     2 (. 
                                                                    Yahoo (www.yahoo.com).
&2 '       +1  2   1& '-' 
     9 +(                  . +(.          '               ' ' &(                 +1  +
  '           &.2           1'             & )'             +1'   
   +       +1  '  )( -' 1  '1                     &( Redis (data structure server),  ) -62
 5       % «[&.$1(».                                0 -   '& 7 (') 1  
                                                                    &.         Redis         & '         2
   7 1 , & '  % ) (   +
                                                                     (   1 & +-  1.- 0, 
    &.2,   ( &2  +  %
                                                                     -) 1) (key-value cache and store), %&




                                                              115
)   -)2  %             ( +  ,               .(,   (    .&.).  '
 '& )(   '& )(  ,  (                      1.         & + 2       (
(, 0, %. 0, '                   2              +1'   & &        2  +.
 ' (in-memory database), )             (, %             (  ( 1   % ItemCF,        9
( &2 ,           &5' '           (,(            &     9           & ) +      & '
 '    2  . 1                             +1  '  &  %)(  , ) 
   %., Redis  9  '+ +                    ( ).  ( 1& ''
&( (snapshotting) 1         2 '                  &   (   '-6: On-line Architecture
92 & ((2 2   rdb),                     (  ( ( -6 1  ( « ») 
   % 12 (append-only file)    +                   Off-line Architecture (  ( (       '-6
    6 1  ' 12. / (                          &(    9).
.  +(   ' Redis ' '-':                         On-line Architecture  -) '  &-6
   &&9 1.2 (transactions) 1  ,                    (   ( (. . 1):
 1.'        &%(        .'- & 
(Publish/Subscribe messaging paradigm), 1 9 +
      '1( Lua (Lua scripting) 
    +1           %          ,
    +1   -)2  %)( 
91 ( -)   ) & '-' 1 ,
)        91),  1.'
1 )(         %  /              16'
 . 0, (LRU eviction of keys),  9
  )         .  + 
  1 (Automatic failover).  9 (
1 ,  +      %
 & Hulu         0, Redis '               4-
 &       1 2   ) '  '
  1 & + + '&  (') 1  
&.                                                                   ". 1 HULU - On-line Architecture
    )         &12       & + 2
( Hulu          ) '          +1             x User profile builder.  ' 9& %     +1  '
1 )( 1           )' '( ( (                             '       & & +(2
  %  & & '   , (                              +,         (2         +1'    & '
 . ,  )    ( ( &'                           '  &2 %     2.
 & )'           +1  2              &(               ( &(            +1-'  & '
, ' ( '   (                              ' ) 1&            (Topic
. 2, & )( 9(, (,                        Model),   & ' %.  &.2.
&)       &2                    +1  +               (   ' %    ) ,
 & )'.        '     9& %             +1  '                0  & '  0  2             
 '       &)2   +, & '-'                         +1' Hadoop  .
% (             +1  2                   &(            x Recommendation Core.  1 &(
 & )'.          B &)2               +                +1  + 2           1 '
    +1  ' &) '' 1                             & +       & )'      9& %
%     2   . +( '.                             +1  '   Hulu. B& )'
    " & +'                   Hulu             (' '-' & ' ) 1&  (topics)
  )  , &               ( 1&):                   (  % (show). B& )'
    6+      +1  '            -62                     +1  2 )( -'    
 . ((') &  + )                     6+              2  &.2 & ' .
  & +.                %         & 9.            x Filtering.  +.'      +1'   & '
 &( &( & '   ( ' '-' &                             ) %    &.2,
 ,   ( 5&'-'            %( (show),                   ) %  Recommendation Core,  . +-
 9 ' (   ' ( &(                                      -)' 9   (  (
    +1  + 2   . [ ( &(                               +1   &).
 & '-   2                   +1  2
   (  ,  9  2%                          x Ranking. "9        +1' & '
( (2         +1  '). ' ( &(                     ),%    )   & )2   %
(  '- & '-6  +,  (                                 +1  '.             &.2,
&()     & '-   2  .-                                )(2  Recommendation Core, ) 
    +1  + 2     2 ( 6(                      +',  1 9'     
                                                                      '&, ) (    +1  +  (  &




                                                           116
      (               6   
       ( &).
 x Explanation. (2  & + %
        (explanation) & ' 9& % 1
     & %(  & + 2  2
         +1  -       . )', )
     & +         2     ' ''    9(
    0    & + 2 (,  
          1 ' & +         '( & '
         +1  ' )( ' '    (
     & 92                  &.2
    ( , «  & 9  +   ,
    ) (     + f»).
  Off-line Architecture  -) '  &-6
  (   ( (. . 2):                                     " 2. HULU - Off-line Architecture
x   Data Center.   +1' & ' ' &(
          +1  + 2    .  '             4  #        
    ' &(       +1-'  '. (
                                                              B +1     Hulu %- %  
    f,  9 Hadoop  .
                                                            ) &( ( ( 1 ,  
x   Related Table Generator.      +1'         400     +                  &     '.),
    &     .          ( -6 (                  1   ( ' '' )
    (        +1  '. /&  .               9 2 1&)2   ) 1'             '
     &9       1 +(         2            1 ' 1.  1 &( 1 ' & +
     +. &%' 1 +(,       )(             ( &(            (       . + %      &
    1 +  1  .                             & '   '  16' % 
x   Topic Model. 7)2 1& (Topic)                2,      ,   (   &
     &9    &), -6      9-                  & '+ & '        (,'   + 
       +.    $ & +                          Hulu )    2           6& & '
     1.  % 16'                         . 7 1 ,  1
    (LDA)     & '      '   1&                    (  & + 2  2 &(
     &96  92  .                              ' '' 9(   1 ' 1
                                                             +      % & Hulu.
x   Feedback    Analyzer.    1       
       +1  2       1 '            )                 1 &(        & .  (' '
      & '+ 2%      &). 7                 1 )( 1  2  & (  )(
      (2    &'  +,                     & %  )()         &      +1  2  ,
     )       +1  2,   1                 ) (      9      (                   
    0       +,  2%                % +. B 1 +  1 &(,
      ) & 9.                             , (' '-' &            %(,   (
                                                               +1-'  +,2          ' +-,  9
x    Report Generator. Š  )      1 '
                                                             %(,   (         +1-'    . 
       + )(,           +1' 1 )(
                                                              0 &(   )            &)
     , )   & & '  1 ) 
                                                           &             +1  2          (( -'
      (  & + 2 (.
                                                            &.         -          (
      & +, )            (               &( ( ,     1 (
Hulu 9      +1   &   & (             ,     '.2   ( )2). 79, 
 () 2     Map      Reduce,      ) -62             0 &(  % (( +'
0   , 1 )( 1&)                    &.    1 (    &'
%.         1     &(,   1 )'          ) .  1   1.    (
 ., , % )'     [1].                 . Œ6 & 2 9 2 1&)2  1
   )   % 2        (  1-62            (  2 &(  ' (' 
 & + MapReduce        +1' Apache Hadoop –              +1  +        & )2             )
 '  1.' MapReduce  ((                       +1(   -  %(            .
 &(  &   '1( Java.                              7  & 1) + % )               +1  2
                                                                    +1    +(  2
                                                                1. &12     
                                                           .     &    0   +, 2
                                                           &%  +-,     %)(       ()  +(
                                                                    2    +- .




                                                     117
    +1       +1  ' & (                             Conference (2009) Volume: 5823, Publisher:
1                 1.                      Springer, Pages: 293-309
2 & '   0  2  (       0                  [2] Konstantin V. Shvachko Apache Hadoop The
1.                                                             Scalability Update []  (2 ].
       1  ( &(  %                           "9 &  : URL: https://www.usenix.org/
& +' ( &(          0    &'                          system/files/login/articles/ 105470-Shvachko.pdf,
 (   2  (                                       2011.
)   0        +1( & &                      [3] Liang Xiang Hulu's Recomendation System
   -    2          6. 79                      []  (2 ]. "9 &  : URL:
 9 (' '+'  + & '  1 )(                              http://tech.hulu.com/blog/2011/09/19/recommenda
&    &' + , & %(                         tion-system/, 2011.
   '  &   ( .'                           [4] Zan Huang, Wingyan Chung, and Hsinchun
 (   2.   ( & '                           Chen. A Graph Model for E-Commerce
1 ' 2 ' ''        ('                             Recommender Systems. // Journal of the
     '(/0 (  &         &.                      American society for information science and
         )(        +1  '.                            technology, 55(3):259-274, 2004.
  ( 1  ' '-' &( e-                       [5] 2 2 ‘   
mail ( ,    2%                                  +,           &(  & + 
( ,     &  ( '                                  & 9 2   // B   +'
 2 )          1 9 +     '+  (                          .' 1 )
    + )  & %(                              (  %9(  Highload++, 2008.
  .
                                                                   [6] $. 7 9  " & +( (:
01                                                                ’+ 1. &           & &(   % (
                                                                       []  (2 ]. "9 &  : URL:
        1 -)   &             +, )                   http://www.ibm.com/developerworks/ru/library/os
 & +( ( ' '-'                 9(                 -recommender1/, 2013.
0    ( &  . B 0                         [7] B            
   (,                 .  %                     & +'  «[&.$1(»
 & 9' ' '' & 2 1 92, 1&)                              []  (2 ]. "9 &  : URL:
   ( : [&.$1(                               http://siliconrus.com/2015/03/yandex-music/, 2015.
(&  ),  , Hulu ( &  ), ..                 [8] Davidson, J., Liebald, B., Liu, J., Nandy, P., Van
 '-       '   )-  , +,                        Vleet, T. The YouTube video recommendation
+1      ) '  )             62                    system // (2010) RecSys'10 - Proceedings of the 4th
 .                                                              ACM Conference on Recommender Systems, pp.
       & 2   &  2 1                             293-296.
  (              & &                     -             [9] Benhardson E. Implementing a Scalable Music
 & +(   9 % 1                            Recommender System []  (2 ].
   (,'          (..    '                       "9 &  : URL: https://www.nada.kth.se/
 9&'        +1  2) & 92                               utbildning/grukth/exjobb/ra pportlistor/2009/
 %  0   2  ..                             rapporter09/bernhardsson_erik_090
 & + 2         (              Hulu               71.pdf – [1. % .  6': 10.05.2015.
 (          ( '             2
   1     +,   &(                            Approaches to Improve the Pertinence of
    +1  + & )'.   +                   Information in the Media Services on the Basis
  +  & (  Spotify, iTunes,                             of Big Data Processing
Pandora, YouTube  Netflix. /& 0 & 
  1 , ..       '-'  9 (                   Stanislav A. Philippov, Victor N. Zakharov, Sergey A.
 &(      Item-based       Collaborative     Filtering,         Stupnikov, Dmitriy Yu. Kovalev
    +1-6                   +1  2       & '
              (         '12        9&                Today, when relevant search methods have almost
 . (           &.,            & +            reached its ceiling, increasing attention is paid to
 & ( () 2 Map Reduce [8-9].                        improving pertinence information. In this paper, the
                                                                  basic approaches to identifying user personal interests,
2 %                                                        as well as detailed principles of the recommendation
                                                                  system known as streaming video service Hulu are
[1] J. Urbani, S. Kotoulas, E. Oren, F. van Harmelen.             considered. This work was supported by the Ministry of
     Scalable Distributed Reasoning using Map Reduce              Education and Science of the Russian Federation. A
     // Proceedings of the International Semantic Web             unique number of work is RFMEFI60414X0139.




                                                            118

</pre>