This paper describes a Shiny application for the R package LOGAN, LOGANShiny. This app was built to provide researchers and education stakeholders an overview of basic tools for starting their analysis of process data from international largescale assessments. Using the log file data from one item displayed at the PISA 2012 creative problem-solving assessment, the app is divided in three modules: (a) Data Preparation, (b) Response times, and (c) Respondent's actions. In each module, the user can interact with the app by analyzing students' performance on the item or comparing specific groups of students (e.g., gender or cross-country analyses). The exploration of such tools can not only illustrate the potential and limitation of process data analysis from these assessments but can also advance one's understanding of how students from 44 countries and economies interact with a problem-solving item on an international survey.
International large-scale assessments have received widespread attention by measuring key cognitive skills and gathering information and data on how individuals use their knowledge in different contexts. For example, since 2012, two important assessments conducted by the Organisation for Economic Cooperation and Development (OECD), the Programme of International Student Assessment (PISA) and the Programme for International Assessment of Adult Competencies (PIAAC), not only started the administration of computer-based formats for a large number of participating countries but also made a number of items with respondent’s log file information publicly available.
These log data contain a record of the interactions between the respondents and the computer testing application during the assessment.
Process data from these kinds of data (e.g., response times and
respondent’s actions) are of potential relevance to researchers and
can provide a better understanding of a range of issues related to
test-taking behavior (e.g., engagement [
To overcome this difficulty, an open-source R package was
developed: LOG file ANalysis in international large-scale
assessments (LOGAN [
To demonstrate the functionalities of the LOGAN package for use
by researchers and education stakeholders interested in process
data analysis, a web-based application using the Shiny app [
To answer this item, students were first presented a stimulus (Figure 2) where they needed to manipulate input variables (top, central, and bottom controls/sliders) to understand how an air conditioner changes the temperature and humidity of a room.
Then, students had to draw arrows on a diagram that represent the relationship between the three controls and the two outputs (temperature and humidity). Full credit was given to students who correctly completed this diagram (i.e., top control impacts temperature and central and bottom controls on humidity).
The available log file data from this item captured the student’s
time on the task, and their exploration on applying and resetting
the input variables using the sliders, the associated temperature
and humidity values, and the state of the diagram at each
exploration. There was no restriction on the number of times a
student could manipulate these features, and they did not change
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
by themselves without the student's interaction. Example of
studies using process data from this item are: [
The following sections of this paper intend to showcase the features of the LOGANShiny app regarding data management and statistical analysis from process data for this item.
In tab “Module 0” from LOGANShiny app, a user will be presented with the particularities of the Climate Control item and its related log-file data. An interactive table displays all the 13 variables and 951,481 entries existing in the data. It represents how students from 44 countries and economies interacted with this item. The description of each variable and the three log events of one student from the United Arab Emirates (code = “ARE”) is illustrated in Table 1. The first log event for the student ID = “04852” indicates when the student was exposed to the item for the first time (event status= “START_ITEM”). For this case, the registered time was 1288.1 seconds since the beginning of the assessment, and no interactions with the item features were recorded (e.g., top_setting= “NULL”).
Less than four seconds after starting the item, this student clicked on the “RESET” button. In this scenario, all the input variables were set as 0 (indicated by a triangle in Figure 2), the output indicated “25”, and no arrows were drawn on the diagram as the default.
About 46 seconds after resetting the task, the same student moved all sliders in one position to the right (i.e., top_setting= “1”, central_setting= “1”, and bottom_setting= “1”) and clicked on “APPLY”. In this case, the temperature value automatically changed to “27” and the humidity to “28”. Again, the status of the diagram was still in its initial setting.
When the student interacted with the diagram, the information displayed in the “diag_state” variable was represented as a binary number (e..g, “'000001“) with each digit associated with one input and one output variable (e.g., top control and temperature). After looking at the information that one can extract from the available log file data, analytical tools are presented at the LOGANShiny app. For example, a summary of the total number of event actions (including "START_ITEM" and "END_ITEM") can be performed in the app. Figure 3 illustrates the log events from 1,015 students from Bulgaria (code=”BGR”). The same analysis can also be done for data from other countries.
From the provided summary statistics, one can verify issues related to the OECD log data. For example, a student from this country has only one entry in the log file data (i.e., one event action). Since it is expected to have at least two events for this dataset ("START_ITEM" and "END_ITEM"), the app emphasizes the importance of a closer look at the data and acknowledges researcher’s freedom to review/filter/delete such inconsistencies. After data management, two analytical tools are provided in the subsequent tabs: Time (Module 1) and Actions - Cognitive related (Module 2).
On this tab, the amount of time students spent on the Climate control item is analyzed. First, a user should decide if the analysis of the total time will be conducted by item performance (CP025Q01=0: incorrect answer; CP025Q01=1: correct answer) or gender (ST04Q01=1: female; ST04Q01=2: male). Later, the user can choose if the analyses will consider all countries or select a specific country, as illustrated in Figure 4. After these choices, two types of descriptive statistics are provided: a summary table and a density plot.
Figure 5 displays the information that one can gather from the LOGAN package for the analysis of the overall time and by item performance. A total of 30,345 students from all PISA 2012 participating countries and economies was analyzed for the Climate control item. The maximum amount of time spent on this item was 26 minutes from a student who got an incorrect answer. For the group of students who got a correct answer, the maximum was 16 minutes.
In general, students spent an average of 2 minutes on the task. However, this estimate is not precise since negative response times were observed in this sample (i.e., the minimum amount of time for those who got an incorrect answer was equal to -0.43). Although one could remove such cases from the dataset as they lower the average values, it is displayed in the LOGANShiny to reveal another inconsistency in this log-file dataset.
Even though this type of discrepancy could possibly be detected at the data preparation stage of analysis, it was left to the “Module 1: Time” tab for the LOGANShiny to highlight again the importance of further inspection of process data and proper data manipulation of the files from large-scale assessments.
Figure 6 illustrates the distribution of time by the performance on the task obtained with the LOGANShiny app. One could also plot the distribution of time by gender.
To illustrate how to explore the actions recorded in the log files,
LOGANShiny describes two respondent’s action strategies
discussed at [
To illustrate how to derive these VOTAT variables from the log data, the third log event from Table 1 shows the case where the student selected top_setting= “1”, central_setting= “1”, and bottom_setting= “1” before clicking on apply. In this scenario, both VOTAT 1 and VOTAT 2 would receive the value “0” where no isolated variation on all the controls were found.
Based on these categories, one can investigate how performance outcomes and VOTAT strategies are related by country, item level performance (CP025Q01=0: incorrect answer; CP025Q01=1: correct answer), and problem-solving overall performance (first plausible value, PV1CPRO). To do this on LOGANShiny, one should select the type of VOTAT strategy they are interested in, followed by each participating country the analyses will be related to (Figure 7): After these choices, two types of descriptive and correlational statistics are provided: a summary report and a frequency plot.
On LOGANShiny, it is possible to conduct a statistical summary of students’ exploration via the “VOTAT 1” strategy and its relationship with performance. This analysis is presented as a report divided in three parts: (1) frequency table, (2) measures of association between strategy and item performance, and (3) summary of test performance (considering the first plausible value from the PISA 2012 problem-solving assessment) by VOTAT strategy. Figures 8 and 9 show an example of this report. From Figure 8, it is possible to see that about half of the students from this sample applied the VOTAT 1 strategy at least one time. For the group of students who got a correct answer in the CP025Q01 item, the majority (12,404 out of 15,076 students) applied this strategy at least once during the item evaluation. Correlational measures (i.e., chi-square statistic and phi coefficient) are also provided to evaluate the strength of the association between these variables.
Imputation methods are used in PISA to generate plausible values
to report students’ overall performance [
Based on the provided statistics, it is possible to note that students who used the VOTAT 1 strategy on the Climate Control item received, on average, more than 100 score points on the PISA 2012 creative problem-solving assessment in contrast to those who did not use this strategy.
In PISA, student’s scores in the assessments are also divided into
proficiency scale levels to provide a substantive meaning of the
overall performance. For PISA 2012 creative problem-solving
assessment, seven levels of proficiency were created where level 1
(358 < PV1CPRO ≤ 423) corresponds to an elementary level of
problem-solving skills and level 6 (PV1CPRO >= 683) the
highest level. A complete description of these levels is presented
in Figure V.2.2 from the OECD report [
In LOGANShiny, these proficiency levels are plotted in relation to the use of the VOTAT strategy. Figure 10 illustrates this relationship. Here, percentages within the categorized proficiency score are provided in parenthesis for each PISA proficiency level. Findings from this analysis indicate that students on the high level of the scale tend to use “VOTAT 1” more than those on the lower levels of the PISA 2012 creative problem-solving proficiency scale.
In this paper, LOGANShiny is presented as an illustrative tool for
showcasing the functionalities of the LOGAN R package
functions for the analysis of process data from international
largescale assessments. Interactive tables and graphical displays
intended to shed light on the potentialities and limitations of the
use of log-file data regarding data management and analysis of
response times and student’s actions. This app can be a valuable
tool to deepen researchers’ and education stakeholder’s
knowledge on the item features and provide insights on students’
cognitive process. The understanding of how process data can be
extracted and analyzed may not only inspire the development of
new item features that could enrich one’s experience with digital
environments, but also has the potential to improve the
assessment’s results by, for instance, incorporating process data
into the scoring procedure [