Our approach to the Log Analysis for Digital Societies (LADS) task of LogCLEF 2009 is to define three different levels of performance: success, failure and strong failure. To investigate the log files we adopt qualitative and quantitative methods. In a more qualitative approach we attempt to identify intercultural differences in user patterns by visualizing the user interactions with The European Library (TEL). On the quantitative level we calculate the relative frequencies of the different performance levels in relation with several aspects e.g. whether a user used the advanced search option or not.
This paper describes an approach to analyze logs from The European Library (TEL) within the LogCLEF track
at the Cross Language Evaluation Forum (CLEF) 2009. Many different approaches have been applied to log file
analysis
Our approach uses the human visual capabilities to find trends and patterns by providing a visualization of user
paths. This is a typical approach taken in data mining
In section 2 we define the meaning of performance in this context followed by a detailed description of our approach of visualizing the sequence of interactions (section 3). The rest of the paper is organized as follows: Section 4 provides a brief description of the algorithm used to process the log files. Section 5 gives some introductory figures to get a first impression of the available data. Sections 6 and 7 present and analyze the results of our experiments and section 8 contains the conclusion and outlook. In our approach to analyze logs from The European Library (TEL) we assume that there are indicators, which suggest that a particular session was successful or not. One such indicator is when the user chose the Available at Library link to view the record in a particular national library interface1. In our judgment this action indicates that the user came across an interesting document according to the query and hence the session was successful. We developed a definition of performance by examining and interpreting the actions recorded in the log files. Therefore it is possible that our operational definition does not include all possible aspects of success. As this study is not meant to produce an ultimate definition of search success, but rather to try out new methodologies for log file analysis, we do not believe that this is a problem in these experiments.
Within this paper we define and evaluate the following three different levels of performance: • • • success - a session is considered to be successful if one of the following actions is carried out at least once: available_at, see_online, option_print, option_save_reference, option_save_session_favorite, option_send_email, service failure - a session is considered to be not successful if none of the actions above is carried out during the session strong failure - a session is considered to be especially not successful if none of the actions above is carried out and the user never uses the possibility of viewing a full record (view_full)
To enable a more qualitative human assessment we visualized the sequence of the individual user interactions with the interface of TEL. Therefore we generated several XML files following the GraphML file format2and adjusted an existing example application for a simple social network visualization3 for the interactive data visualization.
In order to allow the analysis of multilingual search behaviour we created visualizations for German, Spanish, French, British, Italian, Dutch, Polish and US users, as they could be identified from by their IP addresses. In each case we provide four different views4: • • • • frequency - the size of the edges indicates how many times the first action is followed by the second action success - the size of the edges indicates how many sessions that proceed through the second node are successful failure - the size of the edges indicates how many sessions that proceed through the second node are not successful strong failure - the size of the edges indicates how many sessions that proceed through the second node are especially not successful For reasons of large deviations in the counters of different paths the size of the edges is scaled logarithmically. Due to the fact that the XML files are rather large and for reasons of better clarity we do not display the complete graphs. For frequency graphs we show five levels and for the three performance graphs (success, failure and strong failure) we display ten levels. The frequency view for all users (without dividing them by country) cannot be provided, as even five levels exceed the graph size which the tool can show. The three performance graphs for all users show five levels each. Apart from that only paths where the respective counter (success, failure, strong failure) was not equal to zero are displayed. 1 See http://www.uni-hildesheim.de/logclef/ for a more detailed description of the individual user interactions recorded in the log files. 2 http://graphml.graphdrawing.org/ 3The application is available at http://flare.prefuse.org/download 4The interactive visualizations can be found at http://app01.iw.uni-hildesheim.de/logclef/visualizations.zip
We developed a log file analyzing algorithm and applied it to the provided search action log file. The algorithm is implemented in Java using JDBC as API to access the PostgreSQL database containing the log data. In the following we briefly describe our course of action: For reasons of database normalization we first created two additional tables. In the first table the start- and endtime for each individual session is collected. The second table contains all interactions of these sessions. Session identification has been implemented via session identifiers. Logs with the same session ID were regarded as being part of the same session. Since not the whole database could be treated at once, we sampled the log file in 48h intervals. To prevent sessions from being cut in two at the intersection of two intervals we incorporated a 10h overlap for sessions ending in the next interval. Therefore the cut-off length for sessions varies from 10h if the session starts near an intersection of two intervals to 58h if the session starts at the beginning of an interval. However, the chance that a session exceeds 10h is rather small.
Before starting the log analysis data cleaning was performed in order to ease and enhance data analysis. In this regard sessions with only one system interaction were excluded from the database, as the typical search consists at least of two interactions: search and sift through the results. Sessions with missing information on the performed user interaction (missing value) were excluded as well. The original records include several actions that stand for the user calling up the help function (e.g. show_help_help/german/search_s). For simplification of further analysis we decided to combine these actions in the action show_help. The same applies to actions that stand for using the Link to other services link (e.g. service_netherlands). Using this link, we assume, the TEL user gets the possibility to search for the respective record in services like bookshops (e.g. Amazon), search engines (e.g. Google) or other web services (e.g. Wikipedia). These actions were combined in the action service. Data cleaning eliminated 1.93 % of the data entries (not sessions) from the original action log file. Subsequently, the performance of each session according to the definition of performance (see section 2) was determined.
For country recognition we integrated the IP-to-Country Database5 (last updated on June 03 2009). As the last two octets were missing in the logged IP addresses it was not possible to locate all users (24.11% of the sessions could not be clearly assigned).
In the next step we generated the XML files needed for the search path visualization. An example of such an XML graph is shown in figure 3. 5The IP-to-Country Database is available at http://ip-to-country.webhosting.info/ The nodes contain the id of the node, the performed action and the level within the graph. The edges record the source and target node id and the counters for frequency, success, failure or strong failure. To construct the XML graphs we first collected the required information in nine additional tables, one per language and one table including the search path information for all users. This data structure was the input for the generation of four XML graphs for the four different views described in section 3.
In table 1 we provide some descriptive statistics on the occurrence of sessions within the three levels of performance (see section 2).
all de es fr gb it nl pl us total The percentages express the country’s fraction of the whole dataset. For example 1118 successful German sessions represent 4.48% of all successful sessions (24937). It can be seen from table 1 that the percentages do not vary much between the three levels of performance, which suggests that there may not be a lot distinguishable differences between the different user groups. The column entitled total refers to the number of sessions identified and average corresponds to the countries considered.
As already pointed out in section 3 the interactive visualizations can be found and explored online. In this section we present four different search paths: the most frequently used search path (frequency), the most frequently used successful search path (success), the most frequently used unsuccessful search path (failure) and the most frequently used especially unsuccessful search path (str. failure). Table 2 illustrates these different search paths for the first ten interactions.
Since we could not detect striking differences during the qualitative analysis of the user behaviour for different countries we only show the search paths for the group of all users. The success column contains missing values as after level 7 the most successful search path was ambiguous.
At this point we want to report some observations. Firstly the most frequently used and the most unsuccessful search path are identically for six of the nine user groups (British, Dutch and Polish users deviate). If we do not assume that most of the sessions (see also table 1) and the most frequently used search path are not successful, we might have to rethink our operational definition of failure. Maybe there are users that are already satisfied by having the possibility to view a full record (view_full). Maybe some TEL users use the library primarily for informative reasons.
As to the most frequently used search path we have three different search patterns. British, Dutch, Italian and Spanish users act like we have seen before in the case of all users. They submit a query and then view the results. German, French and US users submit a second query after viewing two full records and a third query after viewing again two full records. A possible interpretation of these differences is that we deal with two types of users here. The first type prefers to sift through the list of search results; he submits his query and then examines at least eight documents without rephrasing his query once. The second type, however, prefers rephrasing his queries subsequently; he only examines a few documents of the result list in detail before rephrasing his query. As we do not know what results the users viewed it stays open whether the first type submits more eloquent queries that return better results or whether the second type is more aware of relevant documents according to his query.
1 2 3 4 5 6 7 8 9 10 Polish users differ the most from the other individually considered countries concerning the most frequently used search path. They use the short title list of records (view_brief) in order to view the results and only once view a full record (forth action). Otherwise Polish users seem to belong to the first type of users that does not rephrase his queries much. Nevertheless we compared if Polish users in fact do view_brief more often than users from other countries. Table 3 depicts the absolute frequencies and the percentage fraction of users carrying out the actions view_brief vs. view_full. In this case the percentages express the action’s fraction of all actions (total) carried out by the respective user group where the user viewed results (view_brief + view_full). As can be seen from table 3 indeed Polish users do not view the short title list more often than other countries and moreover perform view_brief less often than view_full (34.1% vs. 65.9%). Our next observation supports the assumption that our operational failure definition might not capture the full picture. The search path in the last column of table 2 shows a sequence of interactions that one would expect if the session is considered to be not successful. Here the users enter a query, the result list does not fulfil their expectations, they enter a new query and so on. Concerning the most frequently used especially unsuccessful search path again we could spot three different search patterns. British, German and US users act exactly like we have seen before in the case of all users. Dutch users after the first search from the simple search form (search_sim) start their further searches from the search form in the results page (search_res), but otherwise also alternately submit a query and do view_brief. We also checked if Dutch users in fact more often perform search_res than other users, but we did not spot any differences. In this context we are not able to give an account of the behaviour of French, Italian, Polish and Spanish users, as they chose a collection from the theme list (col_set_theme) ten times in a row (for Spanish users the search path was ambiguous after level 5). We suspect that this might be due to some technical reason, because it does not make sense to change the collection more than once before any search was carried out.
The last observation refers to the most frequently used successful search path. In this context it is interesting that for all countries but Italy the fourth action is clicking the Available at Library link to view a record in the respective native national library interface. We have not yet found an explanation for this effect, but further research may take this into consideration. For Italian users the most frequently used successful search path is identical with the most frequently used search path, which might be a further reference to Italian users belonging to the first type of users introduced above.
In order to get quantitative results we calculated the relative frequencies of the three levels of performance for three different aspects which are described below: • • • advanced search - the relative frequencies of success, failure and strong failure depending on whether the user uses the advanced search form (search_adv) during the session or not number of interactions - the relative frequencies of success, failure and strong failure depending on the number of interactions with the system session duration - the relative frequencies of success, failure and strong failure depending on the duration of the sessions (calculated for intervals of 5 min.) Except of course for the session duration condition where the relative frequencies are calculated for intervals of five minutes the relative frequencies are calculated for the number of interactions with the system.
In a preliminary step to determine patterns of search performance, we investigated if there is a link between the number of interactions with the system during one session and the level of performance. To address this issue we compared the performance levels for sessions with 2, 3, 4 etc. interactions. We calculated the relative frequencies by dividing the number of sessions (e.g. successful sessions) by the total number of sessions (with e.g. 4 interactions).
Figure 4 shows these relative frequencies for the first 54 interactions of all users. The first 54 interactions were used as cut-off level, because of statistical reasons. Within this interval, the number of sessions per number of interactions amounts to at least 100 sessions.
If we look at figure 4, perhaps not surprisingly, we see that the success curve grows as the number of interactions increases and the strong failure curve declines. Surprisingly, the failure curve stays almost constant at 0.6%. This effect again points to the fact that our operational failure definition might not capture the full picture. As noted earlier some of the 0.6% TEL users might indeed be satisfied by using the library primarily for informative reasons, e.g. by reading the full records. This point of view is also supported by the fact, that this fraction remains almost constant even for 40 and more interactions. In our opinion this long session durations indicate successful sessions, because otherwise the users would have abandoned their search earlier. That is why we included a forth curve to this diagram where we added up the values for success and failure. The curve tends to one with increasing session length. This leads to two possible interpretations. Either people that search long enough will finally be successful or people, who are successful, will search longer. If and which explanation is the right one, would have to be decided in another user test. With respect to the different countries there are no distinguishable differences.
Considering all the sessions 13.39 % of all search actions were started from the advanced search form. The trend of the relative frequencies of the three levels of performance for users that use and users that do not use the advanced search form (search_adv) is qualitatively consistent with the results for the number of interactions condition. Hence we decided only to show a comparison of the relative frequencies of the successful sessions. As already described above, the relative frequencies are calculated by dividing the number of successful sessions (with or without using the advanced search) by the total number of sessions.
As can be seen from this diagram at each time (after the first 20 min.) the proportions of success, failure, strong failure and success plus failure are approximately the same. In other words, after the first 20 minutes the probability of a successful search becomes independent of the session duration. This could reflect the differences in the search speed of different users.
The aim of this study was to experiment with new methods for log files analysis. As a starting point we developed an operational definition of search performance with three different levels of performance. To enable a more qualitative human assessment we visualized the sequence of the individual user interactions with the interface of TEL. Both, the more qualitative analysis of the search path visualizations as well as the more quantitative analysis of the log files have shown inconsistencies within the data which suggest that our operational definition of performance has to be modified in the following way: • • success - a session is considered to be successful if one of the following actions is carried out at least once: available_at, see_online, option_print, option_save_reference, option_save_session_favorite, option_send_email, service, view_full failure - a session is considered to be not successful if none of the actions above is carried out during the session One proposal for an additional success indicator for future research is whether the session ends with a search or not. This could imply that the user could not find the information he/she was looking for and therefore the session could be evaluated as not successful.
During the qualitative analysis of the user path information we observed some differences between users from different countries, e.g. that there seem to exist two prevailing search patterns (cf. most frequently used search path section 6). Whereas one group of users (British, Dutch, Italian, Polish and Spanish users) seems to examine more documents after the first query, another group of users (German, French and US users) seems to rephrase their queries more often. But of course further user tests are required to determine whether the spotted search patterns can serve as categories to differentiate between several types of users.
Although further research is needed to confirm our findings, basically we can say now that it is possible to investigate user performance from log files and that our refined definition of performance at least in the context of TEL users accounts for the user behaviour.