We investigate whether the subscription status of active users of Twitch can be inferred from their activity patterns in the chats of streamers. To enable a diversity of solutions to this problem, this task was advertised as an ECML-PKDD discovery challenge 2020, called Chat Analytics for Twitch (ChAT). Four participants submitted their working prediction models, which were evaluated at our site. The winning approach achieved an F1 score of 0.343, outperforming the baseline by a significant margin. The most salient conclusion that can be drawn at this time is that interaction behavior plays a crucial role in solving this task, meriting further analysis into this direction.
https://events.professor-x.de/dc-ecmlpkdd-2020/
The popularity of game streaming and corresponding platforms, such as Twitch,2
for entertainment and professional e-sports is on the rise. The basic function of
such platforms is to enable a screencast along with commentary to a live
audience. While most streamers focus on streaming and commenting the
gameplay of games they are currently playing, other video and audio content is also
streamed, though at a much smaller volume. The audience, in turn, can interact
with the streamers in a channel’s chat and by other means. This enables
streamers to engage with their audience in real time in order to build a followership
and, eventually, to monetize their channel. The audience can donate to streamers
and subscribe to the channel by paying a monthly fee, which is typically between
Copyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
2 https://www.twitch.tv
five to twenty-five dollars per channel subscription at the time of writing. This
comes with exclusive chat and channel features such as exclusive channel-specific
emotes, which are still or moving images approximately the size of a standard
emoticon. A channel’s earnings are split 50/50 between the streamer and Twitch,
incentivizing both to convert watching users to subscribed users [
In order to enable the task, we provide a large training dataset consisting of
over 400 million public Twitch comments published along this novel task.
Additionally, we constructed a test dataset with certain characteristics that can be
used as both a benchmark for comparison and as a basis for future research and
analysis. The training dataset was provided to participants, who developed their
prediction models at their own site and then deployed their trained model to our
online evaluation platform TIRA [
This paper describes the task and the evaluation data for this challenge, summarizes the submissions, and presents an analysis of their performance. Our contributions are threefold: (1) An original task to predict the subscription status of Twitch users at given channels based on their interactions and chat messages. (2) A training dataset with Twitch chat messages, and a test dataset with characteristics useful for further analysis. (3) An overview and evaluation of the approaches that were submitted as part of the challenge. The remainder of the paper is organized as follows: Section 2 gives an overview on related work. Section 3 introduces the task and the datasets as well as the evaluation metric and the baseline. Section 4 describes the approaches developed by the participants, including the employed features and models. Section 5 analyzes the results of the submitted approaches. 2
As Twitch is one of the most popular streaming platforms for games,
numerous studies have been conducted to understand the use, the impact, and the
challenges of this new form of media. While many publications examine social,
cultural, and economic dynamics of the platform [
Kobs et al. investigate the usefulness of emotes as indicators for the sentiment
of chat messages [
In this work we introduce a novel task: the prediction of the subscription
status of users based on their channel interaction; obviously this is valuable
knowledge that can be used in a subscription recommendation setting. Many
recommendation algorithms, such as collaborative filtering, try to predict whether
a user is interested in an item (in our case a channel) by correlating her
preferences with similar users [
We define the task of subscription prediction for Twitch as follows: Given the chat messages of a user in a channel on Twitch (including metadata such as timestamps and the currently streamed game), predict whether or not the user is subscribed to the channel. We instructed the participants not to augment their models using data other than the training data we supplied. The only exception to this rule is the use of pretrained models or dictionaries of emotes and their text representations. We ensured that the training and test datasets are disjunct in terms of user-channel combinations in order to prevent leakage of ground truth.3 3.1
A large dataset of nearly all publicly available Twitch comments in January 2020 was crawled using Twitch’s official API. Only channels labeled as English were considered. All user-channel combinations for which the subscription status changed during the recorded time period were omitted, i.e., if a user subscribed to or unsubscribed from the channel during January 2020. For each user-channel combination for which at least one comment has been recorded, the following metadata was recorded: – Name of the channel (anonymized). – Name of the commenting user (anonymized). – Whether or not the user is subscribed to the channel. – All public chat messages of the user in the channel in the recorded time period, each containing the timestamp when the user commented, the game that was played in the channel when the user commented in the form of a string label, and the chat comment/message itself.
Many messages contain so-called emotes, i.e., still or moving images
approximately the size of a standard emoticon. Emotes are very popular on Twitch
and are used to express the emotional state of a user while watching [
The crawled dataset was split into training and test datasets. The training dataset has a size of approximately 37 GB. In what follows, we describe the sampling strategy for the test dataset and report on a brief corpus analysis. 3.2
For the test dataset, we sampled 90;000 user-channel combinations with corresponding metadata, resulting in 636;452 messages. To ensure that different user and channel activities are covered in the test dataset, we employed the following sampling procedure. Given the number of messages in the dataset per user and channel, we categorized each user and channel into three activity classes based on the number of comments: 25 % of the users/channels with the lowest message counts are considered to be of low activity, and 25 % with the largest message counts are considered to be of high activity. All other users and channels are considered to be of normal activity. We sampled 10;000 user-channel combinations 3 By mistake, emotes accessible to already subscribed users were not removed; one of the participants exploited this “feature” (without notifying us), rendering their approach infeasible in practice. Nevertheless, it provides for an interesting baseline. channels a user comments in channels a user is subscribed to comments per user in channel comments per user in channel (subscribed) comments per user in channel (not subscribed) comments in a channel Mean for each combination of user and channel activity, yielding 90;000 user-channel combinations in total for the test dataset.
These user-channel combinations were removed from the training dataset. Additionally, for a randomly sampled 50 % of users in the test set, we removed their comments in other channels as well, such that half of the users are not present in the training dataset at all. This allowed us to analyze whether models perform better for already known users, even though no messages in the desired channel were present. 3.3
After removing the test data and the messages of 50 % of the test users in all other channels, the training dataset contained 29;539;420 user-channel combinations and a total of 410;686;442 public Twitch comments. Table 1 overviews key figures of the data. The training dataset has records of 146;537 channels and 7;923;774 users. On average, each user has written comments in fewer than four channels, while being subscribed to an average 1:5 channels. Among the 29 million user-channel combinations in the training data, 2;368;323 (8:02 %) users were subscribed, and 27;171;097 (91:98 %) were not. Subscribed users have a higher mean comment count than non-subscribed users. The high difference of standard deviation between subscribed and not subscribed users can be explained by the use of bots that are not subscribed to channels, but comment very often in order to engage viewers, or to notify users and the streamer about certain events.
Figure 1 depicts the number of messages that a user has sent or a channel has received. Both histograms show exceedingly active users and channels having sent and received an extensive number of comments. The user with the most comments, “streamelements”, is a bot used by many streamers to send notifications about the channel to the chat. The channel that received the most comments, “xqcow”, belongs to a professional gamer and streamer.
Regarding the content, the word cloud in Figure 2 gives an impression of the words used in the comments. Emotes play an important role in Twitch comments, e.g., LUL and PogChamp are heavily used. The Twitch chat is casesensitive and only displays emotes if they are typed correctly; emotes mostly contain capital letters. Therefore, most words in Figure 2 containing capital
0
20,000
40,000
60,000
107
106
105
letters depict emotes; “normal” words are mostly written lower-case. Besides
emotes, ASCII-style emoticons such as :) or :D are popular on Twitch, too. It is
not surprising that gaming and online slang words, such as “u” as a short form
for “you”, “stream”, “play”, and “lol”, are often used. Kobs et al. [
From the 23 registered teams only four submitted their approaches, three of which submitted also a notebook paper describing their approach.4 All approaches rely on certain machine learning methods (see Table 2 for an overview), 4 Given the importance of emotes on Twitch, and for a bit of fun, we asked participants to choose one of the most common Twitch emotes as their team name. but model the input data in different ways. Table 3 gives an overview of the used features, categorized into four groups: (1) stylometric features describing the writing style of the users, (2) user activity features modeling the behavior of the users, (3) channel activity features modeling the behavior of the channels, and (4) interaction features modeling the relationship between a user and a channel. In the following the approaches are reviewed in greater detail.
VoyTECH by Bayer and Zouzias [
CoolStoryBob by Gärtner et al. [
ItsBoshyTime exploits some shortcomings of our dataset. Since subscribers of channels can use channel-specific emotes, the usage of such emotes from the target channel reveals the ground truth about the user in question. While this approach is impractical, it provides as an interesting baseline since not all subscribed users make use of the channel-specific subscriber-only emotes available to them. To extract subscriber emotes from the training data, a dictionary of channels and their emotes was constructed via a heuristic to extract emotes from the messages in the training dataset: If a word begins with a lowercase letter and contains either a capital letter or a number, it is assumed it to be an emote. While most globally available Twitch emotes begin with a capital letter (e.g.
LUL or PogChamp), subscriber emotes have a lower-case prefix based on the username which is usually automatically generated by Twitch.5 Based on this heuristic, an emote list for each channel in the training data is available. If a new user-channel combination is to be predicted, it is checked whether the channel has already been seen. If the channel is unknown, the approach defaults to predicting “not subscribed”; if an emote list for the channel is available, it is matched with the user’s list of used emotes. In case of a match the user is probably subscribed to the respective channel.
StinkyCheese by Loures et al. [
Rank
Team 1 2 3 4
VoyTECH CoolStoryBob ItsBoshyTime
StinkyCheese
Random Baseline hand-crafted features covering all of our four categories, but each less extensively than in the other submissions. The concatenated feature vector is fed through a fully connected layer for classification. In order to handle the large dataset, the training dataset is split into chunks of 100;000 user-channel combinations, and the model trained on one of these chunks. To improve the model, the authors optimize the hyperparameters on a second 100;000 user-channel chunk. 5
The achieved performance of the participants and the random baseline are shown
in Table 4. VoyTECH outperforms the competition by a fair margin.
Relevant Features. A general trend we identify is that activity and interaction
between games, channels, and users are more important than textual features.
Gärtner et al. [
Generalization to Unseen Users. As described in Section 3.2, 50 % of the users in the test set do not appear in the training dataset, enabling an analysis whether there are any differences in prediction performance between known and new users. Table 5 shows the F1 scores for all submissions on the test set, dependent on whether users are or are not part of the training data (Known Users and New Users, respectively). For each approach, except StinkyCheese, users already present in the training data were more often classified correctly than new users. The drop in performance is the largest for VoyTECH, as it relies on many user-centered features. Given only the messages of a user in the target channel and thus missing additional information from the user’s interactions with other channels, the extracted features are less representative. Still, VoyTECH achieves better performance than the other approaches. Results by User and Channel Activity. Table 5 also shows the performance of the submitted approaches based on different channel and user activities, respectively, as defined in Section 3.2. For the most part, it can be said that, the higher the activity of a user or a channel, the better the model can predict the subscription status of a user-channel combination.
A more fine-grained activity analysis can be found in Table 6, considering all combinations of activity classes of users and channels. Again, the performance is mostly best for highly active users and channels and worst for users and channels with low activity. Most extracted features are based on the interaction of users and channels as well as their content. Having few interactions leads to less data and thus less robust features for a given user-channel combination. Ensemble Approaches. Given that the three approaches that rely on different features and classifiers (excluding ItsBoshyTime), it is interesting to explore ensemble classification. We evaluated four different ensembles: 1. Majority vote, where users were classified as subscribed to a channel if at least two approaches say so, 2. An “any” ensemble, which classifies users as subscribed to a channel if at least one approach says so, 3. An “all” ensemble, which classifies users as subscribed to a channel if all approaches say so, and 4. A “ VoyTECH or else” ensemble, which follows the classification of the best-performing approach, VoyTECH, unless both other approaches disagree with it.
All ensembles lead to overall worse F1 scores than the VoyTECH approach by itself. However, as can be expected, the “any” ensemble has a notably higher recall at a lower precision, while the “all” ensemble has higher precision at lower recall. Thus, these ensembles may still be relevant when optimizing for one of these metrics. The full results for all ensembles are given in Table 7. 6
This paper presents the results of the ECML-PKDD ChAT Discovery
Challenge 2020. It outlines the task, the datasets, the approaches, as well as the results
achieved by the submissions. Our analysis of the models covers different user and
channel activity groups, as well as the generalizability towards new users. We are
convinced that there is potential to further improve the predictions—examples:
adding message contents in the winning submission VoyTECH for raising
the model fidelity, or using ideas from StinkyCheese for better predicting
new users. While most approaches work best with highly active users and
channels, the CoolStoryBob seems to work particularly well with normally active
users. Combining their features and ideas into future models may further
improve the prediction quality. In addition, adding Twitch-specific features such as
the sentiment of Twitch comments (e.g., extracted using the technique described
by Kobs et al. [
We thank all participating teams for submitting their models and papers, and the ECML-PKDD organizers for hosting our shared task as a discovery challenge.