Recent research has shown a high correlation between the g factor (or general intelligence factor) and overall performance in video games. This relationship is held not only when playing brain games but also other generalist commercial titles. Unfortunately, these o -the-shelf games do not allow automatic extraction of in-game behavior data. As a result, researchers are often forced to manually register the game sessions metrics, reducing the gathered information and, as a consequence, the results. The aim of our work is to help to improve the data collection process used in those studies by: (1) reimplementing a small subset consisting of three of the games used in these former studies; (2) developing a telemetry system to automate and enhance the recording of in-game user events and variables; and (3) deploying a web platform to conduct an online experiment to collect such data. With the obtained data, we attempt to predict a player's nal score in a given game from truncated play logs (up to a certain point in time) using neural networks and random forest. This later analysis could potentially allow future studies to shorten experiment times, thus increasing the viability of game-based intelligence assessment.
Since the 1980s, research has been conducted on the use of video games to
measure intelligence [
This last study uses a preselected battery of commercial video games and carries out measurement protocols where a human evaluator observes the subject playing and monitors their performance, noting the results: mainly the level reached by the subject in the game over a xed amount of time.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The aim of the work described here is to determine to what extent this process of assessing intelligence through video games can be further improved if we can instrumentalize the games to have a ner measure of the performance in the game. In particular we are interested to see whether we are able to speed up the measurement process through the use of machine learning techniques.
The rest of the article is structured as follows. Next Section summarizes related work on the use of video games for intelligence measurement. Next, we present the games used in our experiments. In Section 4 we describe the telemetry system developed for the games and the data we have collected for each of the 3 games under consideration. In Section 5, we discuss the results of applying neural networks and random forest to predict the outcome with truncated versions of the game traces. Finally, Section 6 proposes future work and concludes the paper. 2
The notion of \intelligence" referred to in this paper is the one proposed by
Gottfredson in [
According to [
We take as our starting point a study already carried out by researchers
in Psychology [
According to [
The execution of the original study consisted in recording, for each of the 10 games involved, the variables that were thought could in uence each of these cognitive factors and to conduct measurements and calculations on them. After completing the games, the participants had to undergo 6 di erent aptitude tests (two for each of the aptitude or cognitive factors considered) and a video game habit questionnaire to normalize the results based on prior game experience.
Performance in the games was correlated with standard measures of each of these factors. The results revealed a correlation value of 0.79 between latent factors representing general intelligence (g ) and general video game performance (gVG ). This nding led to the conclusion that intelligence tests and video games were both backed by shared cognitive processes, and that mind games are not the only genre capable of producing performance measures comparable to standardized intelligence tests. 3
This paper partially replicates the aforementioned study [
For the three selected games, game mechanics and levels required to replicate the original study were developed from scratch using Unity 3DT M . 3.1
Blek [
In each level there is a set of circles or coloured balls that must be picked up by the line as it moves through the screen. To complete a level, the line must (a) Level example with one obstacle and three collectibles. (b) Trace in motion picking up the second collectible. collect all these items before nishing its path by leaving the screen from the top or the bottom. If the player touches one of these target circles while creating the stroke, it immediately comes to life and the user cannot continue drawing. The main game mechanic becomes on the player de ning a short line and anticipating its movement. To add more variety, some special objects are introduced that act as obstacles, always represented as black elements. When the line touches any of them, its progress stops immediately. The line also stops if the user starts drawing a new line while the previous one is still moving on the screen. A nal peculiarity is that the trace "bounces" when it comes into contact with the left and right sides of the level.
As the player advances through the levels, new types of elements appear, such as balls with projectiles, which shoot a set of small colored balls when touched by the stroke. These small balls have the ability to pick up exactly one other colored ball from the level when they touch it, with both of them disappearing in the process (noting here that these projectiles can freely pass through any black obstacle). The game is clearly de ned in a puzzle category, and is a particularly attractive candidate for the evaluation of uid reasoning (Gf). 3.2
The game Edge [
As for the controls and mechanics of the game, in the computer version, the cube is moved using the arrows on the keyboard, or the WASD keys. The cube can climb up steps, but only one at a time and only if there are no obstacles above it. It can also be moved and pushed by moving obstacles or platforms, some of which can be activated with triggers. Other game features include activators that push the cube a number of squares in a particular direction and brittle oor blocks that collapse soon after they are stepped on.
At certain levels the player can shrink and become a mini-cube. This minicube is controlled in the same way and has the added feature that it can climb walls and access places that the normal sized cube could not. The player must revert to the initial size at some point to be able to complete the level. The player dies when it falls over the edge of the scene, after which it respawns at the last checkpoint reached on the map. Unlike the other two games, the mechanics of Unpossible are much simpler: the player rides on the outside of a curved tube in space and tries to hold on as long as possible by dodging all the obstacles they encounter. The forward movement is not controlled by the player, and all they can do is turn left or right on the tube. Every time players hit an obstacle, they die and are \respawned" at the beginning of the corresponding level.
As the game progresses, the sequence of obstacles becomes more complex and demanding, requiring the player to react increasingly faster and concentrate constantly on controlling their movements. Therefore, it is a very promising candidate to measure the processing speed (Gs), and probably also a certain amount of visuospatial capacity (Gv). 4
In order to record the traces of player interaction with our games, a telemetry system was developed in accordance with the following design decisions and fundamental requirements: { The telemetry system must be event-driven in order to: (1) be as general as possible in terms of the kinds of games it is capable of supporting, and (2) be versatile enough to allow the inference of di erent types of metrics in further processing, even those that were not initially considered. We note that condition (1) refers to a design consideration in the infrastructure of the system itself, while (2) is intended to act as a general guideline in selecting which events to send from games via that system. { The events, given the nature of the experiments to be conducted, must have as mandatory elds, at least, identi ers for the game in which they occurred and the user who produced them, a timestamp marking the instant (following the UNIX format) in which they happened, a representative name that di erentiates it from other events in the game, and a dictionary or list of possible additional parameters that may be necessary for the handling and contextualization of these events. { Sending events should be as simple a task as possible from the perspective of the programmer of each game, ideally resembling the way it is used in the case of Unity Analytics. This means that the instrumentalization in the code must be clear and intuitive to use. Furthermore, the corresponding code must be reusable from any additional game implemented in the same engine without additional adaptations.
Di erent available telemetry systems were analysed (Unity, Firebase and Google Analytics) but they were nally rejected because although they provide dashboards to analyse aggregated data, they do not allow to get the raw events for further analysis. So, following the above guidelines, two systems for sending events to the server were developed, one for use from mobile devices/desktop applications, and the other adapted to WebGL. The latter was nally used for experimental data collection.
The events recorded for each game were selected so that the traces produced during a player's session would give a su ciently accurate picture of their behavior and performance during the game. In this way, we chose to record the following signi cant events: { General events: These are common to most games and serve to represent the player's progression and the course of the session (tutorial start, level end, and so on), as well as various frequent events in generalized games, such as the death of a player (player death). { Exclusive events: they were speci c for each game:
Blek : the player touches the screen for the rst time after a static period (first touch), start of a drawing of a stroke (begin drawing), start of the repetition phase of the stroke (begin looping) and collision with an obstacle in the game (black touched).
Edge: collectible prism obtained (got item), new progress mark reached (got checkpoint), and an additional parameter (num moves) at the endof-level events to provide a total of movements made in the level. Unpossible: no additional events were considered, but new parameters were added to the player death events with information about the turns and keystrokes made by the user in each direction. Similarly, new parameters were added to determine the point on the curve where the player died in the corresponding attempt. 5
The question arises as to whether it would be possible to estimate the intelligence of a player through shorter experiments, now that more information is available than in the original study. What we propose is to predict, by means of machine learning algorithms, the nal results of a player in each game. We look only at the data corresponding to the rst minutes of each session. Indeed, if considering a \truncated" experiment we are able to obtain a robust estimate of a player's nal progress, it would be reasonable to assume that it is possible to reduce the duration of the sessions in the experiment without incurring a serious loss of information.
Following this pattern, in each game, we prepare new data from the \raw" events in the database, recording selected variables at certain time intervals. That is, we produce a \timed sequence" comprised of di erent metrics inferred from a given player's event trace. Since these parameters are game dependent and we tested di erent models, we will elaborate on each speci c case: { Blek: Blek's experiment lasts 10 minutes. We process the data from the rst 5 minutes, at 3-second intervals. For each timestamp, we record: the current level, a Boolean variable that indicates whether the player is thinking or not (by \thinking", we refer to any moment/time when the player is not drawing a trace), and the number of curves attempted so far on the current level. { Edge: In the case of Edge, the trial takes up to 12 minutes. We record the rst 6 minutes of data, at 5-second intervals, due to the slower pace at which events take place in this game (checkpoints and collectable prisms are more separated than the distance that could be traveled in 5 seconds). The variables recorded at each timestamp are: the current level, the percentage of checkpoints reached so far in the current level, the percentage of prisms collected so far in the current level, and the cumulative number of deaths so far in the current level. { Unpossible: Of the 5 minutes duration of an experiment session, we collect data from the rst 2.5 minutes, at intervals of 2 seconds. At each timestamp, we recorded the following 5 indicators: the number of deaths accumulated so far, the number of times the player has pressed the left/right arrow from the last death to the moment, and the amount of rotation in each direction from the last death to the present. A special aspect of this game that should be mentioned is that these last variables are only reported to the server when the player dies or the experiment is over, so we don't have their real values in each timestamp. To estimate them, we assume that they have a linear behavior, and we calculate the corresponding proportion at each moment.
The dimensions of the processed data are summarized in Table 1, where the rst column describes the number of individuals that completed the required play time, the second the resulting number of records generated for every individual, computed from the sampling frequency and sampling time (Blek: 5 minutes, timestamp each 3 seconds; Edge: 6 minutes, timestamp each 5 seconds; Unpossible: 2.5 minutes, timestamp each 2 seconds), and the third column the number of values for every record.
individuals timestamps variables Variable to predict
Range
MAE MAE
linear regression mean prediction
Blek no. levels [
We include the results from the best performing models for each game in Tables 3 and 4 . Figures 4 and 5 show the learning curves corresponding to each model. (a) Blek: 2 dense layers of 64 units per layer, with l2(0:001) regularization, without dropout (b) Edge: 2 dense layers of 64 units per layer, without regularization or dropout (c) Unpossible: 2 dense layers of 64 and 32 units per layer, with l2(0:0001) regularization, without dropout
In a general overview, neural networks have greatly improved the reference models we considered, especially in Blek and Unpossible. In Edge the error reduction is not appreciated to a great extent, for the simple reason of it being (a) Blek: 2 GRU layers of 64 units per layer, without regularization or dropout (b) Edge: 2 GRU layers of 64, 32 units per layer, without regularization or dropout (c) Unpossible: 2 GRU layers of 64 units per layer, without regularization or dropout, with 0.0001 learning rate already in a relatively small scale with a lower range. In fact, in the test suite a worse prediction is attained than with the linear regression model. By observing the graphs, all the models found show a fairly similar behavior: the learning curve of the training set decreases rapidly, while the validation remains stable after a few epochs, exhibiting a slight over t. Generally, when the over t is small and the error in the validation stalls, it is a sign that the model does not have su cient expressive capacity. However, testing networks with more layers and/or neurons per layer has not resulted in a signi cant improvement.
Comparing feed-forward network models and recurrent network models, we spot no clear advantage in the use of the latter, which are theoretically better suited for serial data. Only a slight improvement is shown in the case of Blek. However, due to the high variance obtained on the test set, we cannot con dently state the dominance of recurrent networks. As a matter of fact, this large variance may indicate that we are dealing with a much more complicated and heterogeneous dataset than that of the other cases, or that the variables we included in the analysis do not facilitate the prediction.
Another general phenomenon is that all models perform worse on test data,
having achieved quite promising results on the validation set. This di erence is
notably greater in Blek, whose error of around 1 on the validation data in the
best models can reach a value of almost 5 on the test set. This problem may
be due to the limited number of observations we have. In this case, the random
division of the data into three subsets can greatly a ect the results obtained.
For example, in a su ciently large dataset, the divisions follow roughly the
same distribution, presenting similar characteristics. Conversely, splits in a small
data set may contain very di erent patterns that models are not able to learn
just by looking at the training data. Therefore, instead of delving further into
di erent neural network con gurations, we turn to traditional machine learning
techniques, which are known to perform better on a small dataset. The option
we chose is the random forest [
We follow the same train-test split percentage and employ K-fold crossvalidation. In the same way as the previous case, we tuned a subset of hyperparameters, including: the number of samples drawn to train each base estimator (tree) in the bagging method; the number of variables to consider in each split; the maximum depth of the trees; and the number of trees in the forest. The obtained results are presented in Table 5.
validation MAE test MAE
We observe that random forest models behave similarly or even better than neural network models. We can conclude that, with a dataset as small as this one, it is not necessary to resort to such complex deep learning techniques. 6
In this paper we have presented preliminary results on the application of telemetry in video games to speed-up the measure of intelligence which has been previously demonstrated to correlate with performance in those games. Despite the dataset modest size, we were able to predict a player's nal score in a given game from truncated play logs using neural networks and random forests. By avoiding the need for a human evaluator to collect data in such experiments we could potentially allow future studies to shorten experiment times, thus increasing the viability of game-based intelligence assessment. In the future, we plan to carry out further experiments in order to further validate this automatic approach.
In addition to the application of this approach to intelligence assessment, we envision the use of similar techniques for the dynamic adjustment of game di culty. Since we are able to predict game performance, we can adjust the game based on the predicted performance of a particular player.