This paper presents an empirical study about the temporal patterns characterizing the requests submitted by users to Wikipedia. The study is based on the analysis of the log lines registered by the Wikimedia Foundation Squid servers after having sent the appropriate content in response to users' requests. The analysis has been conducted regarding the ten most visited editions of Wikipedia and has involved more than 14,000 million log lines corresponding to the traffic of the entire year 2009. The conducted methodology has mainly consisted in the parsing and filtering of users' requests according to the study directives. As a result, relevant information fields have been finally stored in a database for persistence and further characterization. In thia way, we, first assessed, whether the traffic to Wikipedia could serve as a reliable estimator of the overall traffic to all the Wikimedia Foundation projects. Our subsequent analysis of the temporal evolutions corresponding to the different types of requests to Wikipedia revealed interesting differences and similarities among them that can be related to the users' attention to the Encyclopedia. In addition, we have performed separated characterizations of each Wikipedia edition to compare their respective evolutions over time.
Wikipedia continues to be an absolute success and stands as the most relevant wikibased platform. It provides a rich set of contents belonging to every knowledge area that are offered in different formats that range from text to multimedia resources. In addition, the Wikipedia’s supporting paradigm, which is based on individuals collaboration and joint of efforts to produce and contribute pieces of knowledge that will remain available for the whole community. The consolidation of Wikipedia as a reference tool and a platform for mass collaboration is endorsed by the increasing number of visits to its portal. In fact, the Wikipedia domain remains within the six most visited ones all over the Internet.
Wikipedia is divided in 268 1 editions corresponding each to a different language and its overall relevance can be simply measured in terms of the number of visits it receives. Currently, the overall set of Wikipedias editions are receiving approximately 13,500 million visits a month. This constitutes an absolute challenge in terms of management of requests and content delivery. On the other hand, Wikipedia organizes the information it offers in encyclopedic entries commonly referred as articles. At the moment of writing this paper, the different Wikipedia editions add up to almost 18 million articles and this number does not stop growing.
As a result of this relevance, Wikipedia has evolved into a subject of increasing
interest for researchers [
Therefore, this paper presents an empirical study encompassing a temporal characterization that may help to describe the evolution over time of users’ interactions with Wikipedia. Furthermore, we will compare the results obtained for the different editions in order to analyze the main differences and similarities among them.
Our analysis focuses on the most relevant Wikipedia editions in terms of their volumes of articles and number of traffic. In addition, the period of time considered correspond to a whole year (2009). Our main data source consists in users’ requests to Wikipedia previously stored by special servers deployed to deal with the incoming traffic. Information about each individual request is registered in the form of a log line whose fields are processed by an ad-hoc developed application. This application filters the requests considered of interest for our analysis and stores its information elements into a database for further examinations.
The rest of the paper is structured as follows: first of all, we describe the data sources used in our analysis as well as the methodology followed to conduct our work. After this, we present our results and, finally, we present our conclusions and propose some ideas for further work. 1 http://stats.wikimedia.org/EN/Sitemap.htm
This section aims to describe the information sources involved in our study and used as the main data feeding to perform our analysis. The visits to Wikipedia, in a similar way to any other Internet site, are issued in the form of URLs sent from the users’ browsers. These URL’s are registered by the Wikimedia Foundation Squid servers in the form of log lines after serving the requested content.
Therefore, the following sections present the principal aspects related to how the Squid log lines used in this analysis are registered, their way to our storage systems and the most important information elements that they contain. 2.1 The Wikimedia Foundation Squid subsystem Squid servers are usually used to perform web caching working as proxy servers. In this way, they can cache the contents browsed by a group of users to make them available for further requests. This results in an important decrease of the bandwidth consumption and in a more efficient use of the network resources. Furthermore, Squid servers may be used to speed up web servers by caching the contents requested repeatedly to them. Under this approach, Squid servers are said to work as reverse proxy servers because they try to reply to the received requests using the cached contents, what reduces, if so, the workload of both the web and database servers placed behind them.
The Squid operation is based on web caching and, hence, it is aimed to avoid the participation of the other database and web server systems in operations for serving requested contents. In this way, when a requested page can be found on a Squid server and it is up-to-date, the page is directly served from the Squid and neither the database server nor the web server have to be involved in the delivery process. Otherwise, the request is sent to the web servers which elaborate the corresponding HTML code and submit it to the Squid for its caching and final delivery to the user.
As the Wikimedia Foundation maintains several wiki-based projects, such us Wikipedia, Wikiversity or Wikiquote, the Squid layers have to deal with all the traffic directed to these projects. As a part of their job, Squid systems do log information about every request they serve whether the corresponding contents stem from their caches or, on the contrary, are provided by the web servers. In the end, Squid systems register a log line with different kind of information for each served request and these lines can be written to a file or sent to another process through a pipe as in the case of the Wikimedia Foundation.
Each log line from a Wikimedia Squid server corresponds to a served user request and constitute a really valuable feed because, among several other information, it includes the URLs submitted by the user along with the date at witch the corresponding content was sent in response.
The analysis presented here is based on a sample of the Wikimedia Foundation Squid log lines corresponding to the entire year 2009. The sampling factor used has been 1/100, so this study has included the characterization of the 1% of the overall traffic directed to all the projects maintained by the Wikimedia Foundation during the whole year 2009. In general terms, more than 14,000 million log lines have been parsed and filtered for this study.
This analysis has focused just on the traffic directed to the Wikipedia project and to ensure that the study involved mature and highly active language editions, only the requests corresponding to the ten most visited ones have been considered. These editions are the German, English, Spanish, French, Italian, Japanese, Dutch, Polish, Portuguese and Russian ones.
Once the log lines from the Wikimedia Foundation Squid systems have been received in our facilities and conveniently stored, they become ready to be analysed by the tool developed for this aim: The WikiSquilter project. The analysis consists on a characterization based on a parsing process to extract the relevant elements of information prior to a filtering one according to the study directives. As a result of both processes, necessary data to conduct a characterization are obtained and stored in a relational database for further analysis.
The lines received from the Wikimedia Foundation offer a valuable information source but they do not include specific information elements to describe certain characteristics of the corresponding requests. However, these elements can be obtained from the URL embedded in each line which, therefore, has to be parsed looking for specific data serving as characterization elements.
More in the detail, the application parser is devoted to determine the following information elements: 1. The Wikimedia Foundation project, such us Wikipedia, Wiktionary or Wikiquote, to which the URL is directed. 2. The corresponding language edition of the project. 3. When the url requests an article, its namespace. 4. The action (edit, submit, history review...) requested by the user (if any). 5. If the URL corresponds to a search request, the searched topic 6. The title of every requested article or user page name.
The parsing process relies on the use of regular expressions for verifying whether an URL, or a part of it, matches a given pattern. If so, its components can be obtained using common functions for string manipulation. On the other side, the filter process consists in assessing whether an URL has to be considered significant for our analysis according to its directives. This is accomplished by checking whether the information elements it contains, once parsed, has been indicated to be filtered.
The application has been designed and developed according to the principles of efficiency, robustness and accuracy. However, flexibility and extensibility guidelines have been also strictly followed. Efficiency has been achieved through several elements such as multithreaded design and filter’s O(1) complexity derived of the use of hash tables. The application robustness has been implemented by URL malformation detection and preprocessing to avoid non-appropriate characters and has allowed to rightly process all the log lines to be analyzed. Flexibility makes the application ready to be used with any sort of log files just specifying in a XML log files the elements to be parsed and filtered. The software architecture of the application allows to easily include new services that can even involve new data to be processed, so extensibility has been also considered. 4 Analysis and results In the following we are presenting our most important results about the temporal characterization of users’ requests submitted to Wikipedia. First of all, we analyze if the traffic to Wikipedia can reliably model the overall traffic to the Wikimedia Foundation. After this, we compare the evolution of the different types of requests over time. Concerning this topic, we will present the different patterns found, paying special attention to the ones showing repetitive schemes. This examination has been specially conducted under a comparative approach to determine whether or not the same tendencies are maintained in every considered Wikipedia edition. Finally, our analysis allow to obtain valuable information about the ratios corresponding to the different types of requests that is also presented. 4.1 The traffic to Wikipedia as a model of the traffic to the
Wikimedia Foundation 2 http://stats.wikimedia.org/EN/TablesPageViewsMonthly.htm 7 ida .20e0+ e p i k i W o t c iffraT 70e+ 5 . 1
Temporal evolution of the traffic 2.0e+07 2.5e+07 3.0e+07 3.5e+07 4.0e+07 4.5e+07
5.0e+07
Evolution of the number of submit operations during the whole year 2009 Jul
Jul Time Jan Feb Mar Apr May Jun Jul
Aug
Sep
Oct
Nov
Dec 4.2 Temporal evolution of the different types of requests to
Wikipedia If we separate the requests to Wikipedia according to their types, Figures 3 and 4 show how each one of them evolves throughout the entire year 2009. We are considering a visit to an article as its page request for reading and without involving any other action. In turn, edit operations are intended as modifications over the content of articles that are finally saved to the database. The difference between edit requests and edit operations is that the first are issued when users just click on the ”edit“ tab placed on top of the articles’ pages whereas the latter are generated when users indicate a write operation to the database to save their changes or their contributed contents. Submit operations are those directed to preview the result of the modifications performed on the current content of an article or to highlight the differences introduced by a given edit operation in curse. History requests present the different revisions (edit operations) performed on an article’s content and leading to its actual version and state.
According to Figures 3 and 4, only those URLs involving visits, searches and edit requests would exhibit temporal repetitive patterns. On the other hand, requests consisting in edits (save operations), history reviews or submits for previewing contents would not present such cyclical evolutions over time. This is likely due to the fact that the requests exhibiting repetitive behaviors correspond to the most usual or generalized types of requests that compose the traffic to Wikipedia. The other kind
Evolution of the number of visits across the year
Jan
Feb
Mar
Apr
May
Jun
Aug
Sep
Oct
Nov
Dec of requests, on the contrary, have a more specialized character and because of this they appear rarely in the traffic. As a result, the most common requests follow the same periodical evolution than the general traffic to Wikipedia whereas the rest of requests show a more spurious behavior.
We undertake now the same analysis focusing on every whole week during 2009. The aim is to determine whether there are patterns involving any type of requests that are repeated along every week of the year. This is done, for example, in Figure 5 for the German, English, Spanish and French Wikipedias. This closer perspective confirms the similar weekly evolution of visits, searches and edit requests in contrast to the spurious and irregular nature of the requests consisting in edit operations, history and submits.
We decided to undertake the study of the evolution of visits and edits at the level of the days of the week in the aim of finding a meaningful closeness between their two temporal variations. As a result of such kind of analysis, Figure 6 presents the evolution of both types of requests throughout the days of the week for all the considered Wikipedias. Visits and edits, in each Wikipedia edition, correspond to the overall year and have been grouped by their day of issue. So, Figure 6 presents their compared progressions and shows a considerably closeness in the evolution of both types of requests in several Wikipedias. Nevertheless, the number of edits tends to raise in weekends for a group of them (French, Japanese, Dutch and Polish). That could mean that, in those editions, editors are not part of the great mass of people visiting the articles but just a minor group devoted to contribute or to maintain them. Requests throughout all the whole weeks of 2009 (DE)
Requests throughout all the whole weeks of 2009 (EN)
Search 5 0 + ts 1e seu q e frro 03 e e+ b m 1 u N 4 0 + tsse 5e3 frrqeuobe 5e02+ uNm e50+ 1 0 + e 5 5 Comparing the number and temporal evolution of the actions requested to Wikipedia Weekly number of visits and edit operations (DE)
Weekly number of visits and edit operations (EN) 6 Conclusions and further work We can extract several conclusions after our efforts for characterizing temporarily the requests submitted to Wikipedia. First of all, we have shown how temporal information related to users’ requests can be obtained from log lines stored by Wikimedia Foundation’s Squid servers. Using this information we have modeled the variations over time of the different kind of requests submitted by users to Wikipedia. Our first finding was the fact that requests to Wikipedia temporarily model the overall traffic to all the Wikimedia Foundation projects. Of course it was what we were expecting, as Wikipedia is, by far, the most trending project maintained by the Wikimedia Foundation. However, we managed to obtain a high degree of correlation between Wikipedia’s traffic and the requests directed to all the Wikimedia Foundation projects. In addition, we have illustrated how demands to Wikipedia consisting in visits, searches and edit requests present repeated patterns over time as they are the most generally solicited. On the other hand, submit or history requests and edits present a spurious and irregular nature because of their most specific character. In relation to this topic, the size of the sample may be determinant as the low percentage of edits contained in it can prevent the observation of cyclical distribution. So, further examinations should involve higher sampling factor to accurately analyze the presence of stationarity in the distribution over time of edits.
)( 80 ioedoepog 0 l 00 . 0 t 0 f 40 r 7 b m uN Sun
l)( 1060 frdpoooeeg 00 .
12 it Sun 10 ubN
m Sun Sun 0 l.()p og doeo 750 it
f 6500 rbe m uN l) og 00 .(p
ode 46 it 0 f 430 rebo m uN 0 l.()p
og 08 fdoeo 80 it 820 rbe m uN )
) g
g s
s e
e u
u q
q f
f o
o r
r e
e b
b m
m u
u N
N 6
6 0
0 +
+ e
e 2
2 )
) g
g 5
5 0
0 s
s +
+ e
e e
e u
u 2
2 q
q f
f o
o r
r 4
4 e
e 0
0 b
b +
+ e
e m
m 2
2 u
u N
N 3
3 0
0 +
+ e
e 2 2 Monthly number of actions (DE)
Monthly number of actions (EN)
Monthly number of actions (IT) Monthly number of actions (JA)
Jan
Mar
May
Jul
Sep
Nov Jan
Mar
May
Jul
Sep
Nov
DE Monthly number of actions (ES) EN Monthly number of actions (FR)
Search req. o l ( s t 3 0 + e 2 6 0 + e 2 5 0 + e 2 4 0 + e 2 3 0 + e 2 6 0 + e 2 5 0 + e 2 4 0 + e 2 3 0 + e 2 6 0 + e 5 5 0 + e 5 4 0 + e 5 3 0 + e 5 Jan
Mar
May
Jul
Sep
Nov
Jan
Mar
May
Jul
Sep
Nov
Though the quantitative analysis of requests to Wikipedia may be considered tangential to the main aim of this paper, we consider that some observed findings in this area deserve to be mentioned. In this way, we have been able to appreciate how searches and edits are, respectively, the most and the least requested types of actions. Interestingly, we have shown how there is a significant relevance between the number of edit requests and the writes operation to the database that indicates that edit requests are abandoned by users in a considerably number of times. On the other hand, edits and submit requests remains very similar in number, which means that users usually exhibit the adequate habit of previewing changes before applying them to be permanent.
In the future, we plan to add geolocation to the temporal characterization process. In this way, a reference time plus the geographical position could better serve to determine the habits of the different communities of users when browsing Wikipedia. Furthermore, a closer analysis of the evolution of the different types of requests will allow to find more accurately defined relationships among them.