In this paper, we report the process of building a binary classi er for the Chat Analytics for Twitch challenge. The goal is to predict the subscription status of a user in a channel based on their comments. As a result for this task, our nal submission achieves an F1-score of 0:1422 on the dataset used to evaluate the classi ers. We explore the results and try to understand which scenarios contribute to this performance. We hope these analyses can be used to direct new research and to improve classi ers designed for this or related tasks.
Streaming platforms such as Twitch are extremely popular, attracting millions of broadcasters and hundreds of millions of viewers each month2. Through them, multimedia content creators stream a multitude of recordings or live streams on the internet. Besides that, some platforms enable live interactions between the broadcasters and their audience. This is the case of Twitch3, which started mostly as a game streaming platform and has since grown to include other types of content, such as cooking, programming, music, and tourism channels.
To encourage and attract content creators, the Twitch platform has a
monetization model in partnership with the owners of stream channels [
In this context, the Chat Analytics for Twitch (ChAT) challenge [
The ChAT dataset was provided by the ECML-PKDD 2020 Discovery
Challenge [
Before proposing a classi cation model, we made some initial exploratory analysis on the dataset. We generated a random sample containing 500; 000 userchannel pairs, of which 85; 985 are subscribed and 414; 015 are non-subscribed. First, we measured the number of messages per user-channel pair. On average, users write 70 messages in channels they are subscribed to (median 20). On the other hand, non-subscribed users only write an average of 25 messages (median 4). A similar trend can be observed for the total length of the messages. In other words, users who are subscribed to a channel tend to post longer messages than non-subscribed users.
We also analyzed the frequency of messages for subscribed and non-subscribed
users. Figure 1 illustrates the timestamp of consecutive comments (starting with
a timestamp 0 for the rst comment) for two example instances in the same
channel, one of a subscribed user and another for a non-subscribed user. Note that,
while the non-subscribed user made more comments, that user's participation
had a lower duration than the participation of the subscribed user. While this
single example cannot be used as a generalization, it showcases the intricacies
of trying to relate frequency of comments to the subscription status.
Deep learning architectures have become the state of the art solution for several
tasks, including text classi cation [
Text Input Embedding
Features LSTM
Concatenation
Dense
Output
4 Implementation available at https://github.com/lourestc/chatpkdd
the messages in the user-channel pair. The word embeddings were randomly
initialized, with a size of 100, and then learned during the training process together
with the remainder of the network. We avoided using pre-trained embeddings,
given that they are usually built on more formal texts (such as Google News [
The text representation learned by the LSTM is concatenated with additional features, which are described in the following subsection. These combined tensors are then run through a simple dense layer, and the output is predicted with a sigmoid activation function. The loss of the model is calculated with the Binary Cross Entropy function. 3.1
Considering the subject of the challenge, the data presents other relevant information that could be useful apart from the text of the messages themselves, such as the user-channel relationships and the structure of the users' participation in the chat system. Thus, besides the text model previously described, we de ned features that could help the classi cation model based on these conversational traits. For each instance, features were calculated based on these characteristics, taking into account the other users and channels in the dataset.
These features can be divided in groups related with three di erent aspects: verbosity, which is derived from traits of the user's messages in the given channel; attendance, which is related to the frequency in which the user post messages in the channel; participation, which indicates how much the given user and channel participate in the dataset as a whole.
Verbosity. These features measure how much content is generated by the user in the channel. Both the amount and the length of the comments are considered. As a reference point to these numbers, average values for the user and for the channel are used as additional features.
{ NumMsg UC: The number of messages sent by the user in the channel. { LenMsgs UC: The text size (term count) of the user's concatenated messages in the channel. { AvgLenMsgs U: The average text size (term count) of the user's concatenated messages, considering all the channels that they participate. { AvgLenMsgs C: The average concatenated messages text size (term count) of users participating in the channel. 5 For examples, see https://twitchemotes.com/ Attendance. These features aim to capture the assiduity and permanence of the user in the channel. They measure both the length of time over which the user participated in the channel, as well as the intervals between messages. { AttendanceTime UC: The total time of user's activities in the channel.
In other words, the time between the user's last and rst messages in the channel. { AvgInterval UC: The average interval time between messages sent by a user in the channel. { StdInterval UC: The standard deviation of the interval time between messages sent by a user in the channel. { MinInterval UC: The shortest interval time between messages sent by a user in the channel. { MaxInterval UC: The longest interval time between messages sent by a user in the channel.
Participation. These features inform the general involvement of the user and of the channel in the dataset. In a way, they measure the dedication between the user-channel pair.
{ NumChannels U: The number of channels in which the user participates. { NumUsers C: The number of users participating in the channel. 4
To run our training process, we used a computer with an Intel(R) Core(TM) i79700KF CPU @ 3.60GHz processor, 56 GiB of memory, and a NVIDIA GeForce RTX 2080 Ti GPU. The implementation was done in Python, using the Keras package. 4.1
For the training process performed in this project, we divided the training dataset into les of 100; 000 (one hundred thousand) instances (channel-user pairs) each. All preprocessing and dataset manipulation steps were performed on a single sub le at a time. So, for example, when counting the number of channels a user participated in, our experimentation only considered events in the same sub le of the instance for which the feature was being calculated. Although this does reduce the overall quality of the training process, this was done so that the preprocessing steps needed for the model described in Section 3 would more easily be performed.
In this paper, only one such le was used for the training step of the model, and a second one for the validation of the model. Using the remaining instances would potentially provide a more robust result in future works.
As a rst step, we preprocessed the dataset into a format that could be more easily used in our model. The messages from each user-channel pair are initially concatenated in order of occurrence, with no separation tokens between messages. Then stop words are removed from the text. We split the text by terms, considering each sequence of punctuation characters as a single token (in order to capture the use of simple emoticons, such as ;-/). Finally, we calculated the features de ned in Section 3.1.
This preprocessing step took 11 minutes to be performed for the le we used in the experiments. 4.3
In this section, we describe an ablation study performed over a number of aspects that may in uence the performance of our model. The hyperparameters were selected using a second 100:000-instance le as the validation (dev) set, and the scores are also reported in this set.
We train a number of models with di ering hyperparameters, exploring the
impact of the optimizer, the learning rate and the maximum number of words
on the F1-score. We follow practical recommendations for some of the most
commonly used hyperparameters: it is often the case that a few of them make a
big di erence [
We rst start experimenting on the learning rate, perhaps the most important hyperparameter [3, p. 424]. It is typical to pick the learning rate using a grid search with values approximately following a logarithmic scale [3, p. 428], so we take values within the set f0:1; 0:01; 10 3; 10 4g.
Regarding the optimization process, there is no consensus on the optimal
algorithm to use [
Lastly, we change the number of words per sequence used for the text representation, with 200, 300 and 400 as values. These values were chosen based on the computational resources available at the time. Note that even with a 400 value, this limit is not enough to cover all instances of test set and training set, which have 865 and 9120 instances, respectively, that can not be completely covered with 400 tokens. Moreover, cutting out the last words of each sequence can have a di erent e ect than cutting the rst or at random, although we have not evaluated these di erent approaches in this work.
The best overall result was obtained with the Adagrad optimizer, learning rate of 10 4 and 200-word representation; the most promising experiments are shown in Table 1. Once the best hyperparameter selection was de ned, we performed a nal training run on the model over 10 epochs. This execution took 83 minutes and 13 seconds to complete. 5
For the ChAT challenge, participants were asked to submit their code to run on
the TIRA platform [
Our nal submission achieves an F1-score of 0:1422 on the test set, ranking in fourth place in the challenge. As a baseline, a random sampler classi er achieves an F1-score of 0:0741 on the same set. 5.2
In this section, we analyze how our model behaves based on the activity level of the user, provided in the ground truth le of the test set. This activity is classi ed into low, normal or high, according to the number of messages written or received.
As shown in Figure 3a, the model predicts with 0:72 accuracy the subscribed users with high activity level. On the other hand, it makes wrong predictions for subscribed users with low activity, achieving only 0:232 accuracy for this subset. This tendency, however, is not seen among non-subscribed users. While the model has high accuracy (0:787) for the prediction of low-activity users in this scenario, it often fails (0:38 accuracy) for the highly active ones, as seen in Figure 3b.
This analysis indicates that our model tends to correlate activity level with the probability of the user being subscribed, as we often correctly predict highactivity users that are subscribed and low-activity users that are non-subscribed. Considering the features we employed, as described in Section 3.1, this matches the expected capabilities and bias of the model.
We also observed how the model performs with known and unknown channels and users. While the model performed better with channels from the test set that were already seen in the sampled training set, with an F1 score of 0:1964, 6 See results in: https://events.professor-x.de/dc-ecmlpkdd-2020/results.html - Site visited on July 20, 2020
(a) Subscribed users
(b) Non-subscribed users it performed worse with seen users, having an F1 score of 0:0962. It is worth noting that, as known users and channels consist of a small portion of the dataset (1; 474 known channels and 2; 255 known users, out of the 90; 000 test instances), inferences based on this data can be unreliable.
In order to better visualize the di erent instances in which the StinkyCheese model was able to correctly predict the subscription status or not, we present a few examples of messages from user-channel pairs. From these examples, we can see that the task of identifying if a user is subscribed or not based on the chat is a complex one. Subscribed users can have interactions in chat as short as those of non-subscribed users, while non-subscribed can have conversations that seem as intimate as a dedicated subscriber.
True Positive (subscribed, predicted as such): { \so |you getting a new monitor" { \Your welcome. I nally get to catch your stream |Ok. Haven't played d2 in 3 weeks. Content wasn't keeping interested |I hope season is good. This season was lackluster"
{ \Hey |Dude |Wassaup |Wanna play" { \bots |cya and happy new year |the beer was perfect"
{ \just stopping by for a minute to whish max and everyone a happy new year" { \!giveaway |What do u play on |Pc nvm"
This work seeks to identify the subscription status of a user in a Twitch channel with word vectors fed into a LSTM and then concatenated with conversational features. Using chat messages for machine learning tasks is a complex endeavor. Not only is the textual content of the messages an important factor, but the structure of the conversation and the connection between agents also feature into the picture. Studies in the eld have several possible angles to look into, and there's still too much to improve.
Analyzing our results, it is possible to observe that the prediction of our classi er is attached with the user activity, so it tends to predict as subscribed a user that has high activity in channel, and to predict as non-subscribed an use that has low activity in channel. In order to make the model better at dealing with other cases, increasing the complexity of later steps of the network could help the model to extract more complex relations between features and make more accurate predictions.
We also have other limitations, such as the fact that we use features that heavily summarize complex information regarding the structure of the conversation and user-channel relationships. A more complex network structure could be used with the extra metadata provided in the dataset, for example, using the timestamp sequence as a whole instead of simply taking measures over them as features.
As future work, we also intend to train our model again using pre-trained
word embeddings created with Twitch reviews, as proposed by [
In addition, some features have a larger range of values in comparison to others, such as those based on timestamps in comparison to those based on message length. This can disrupt the classi er and a ect the results. To avoid this problem, the feature values could be normalized, potentially improving the learning process.
One nal point that could greatly improve the quality of the model would be a heavier training process. This includes both the use of the entire training dataset and the improvement of the hyperparameter search.