In recent years the number of players that are willing to submit a video game review has increased drastically. This is due to a combination of factors such as the raw increase of video gamers and the wide use of gaming platforms that facilitate the review submission process. The vast data produced by reviewers make extracting actionable knowledge dificult, both for companies and other players, especially if the extraction is to be completed in a timely and eficient manner. In this paper we experiment with a game review summarization pipeline that aims to automatically produce review summaries through aspect identification and sentiment analysis. We build upon early experiments on the feasibility of evaluation for the task, designing and performing the first evaluation of its kind. Thus, we apply variants of a main analysis pipeline on an appropriate dataset, studying the results to better understand possible future directions. To this end, we propose and implement an evaluation procedure regarding the produced summaries, creating a benchmark setting for future works on game review summarization.
• Applied computing → Computer games; • Computing methodologies → Information extraction. summarization, natural language processing, sentiment analysis, game reviews, Steam.
The ever-expanding popularity of digital games is evidenced by the
large profit margins of the commercial game industry sector [
We explore the topic of summarization for game reviews using a large dataset of Steam reviews from 12 selected games. The goal of the summarization pipeline is to extract users’ views on diferent facets of games such as graphics, audio, and gameplay [28], leveraging textual sentiment analysis to identify and positive and negative review snippets, creating a composite summary of indicative comments on a specific game facet. Unlike the numerical aggregation of Metacritic or Steam, this approach extracts individual sentences (and criticisms) contained within a usually dense review and attempts to classify those in terms of positive or negative automatically (rather than based on the user’s binary recommendation). The presentation of the game’s summary, which is split based on diferent aspects typically criticized in games, can be valuable for both players and designers. For players, the statistics derived from this process (e.g. ratio of positive versus negative comments in one aspect) can act as an expanded game scoring system not unlike professional game reviews which gave a score to graphics, audio etc. For designers, the indicative comments split per sentiment and aspect allows for a quick monitoring of players’ current favorite features. Moreover, the flexible way in which aspects are defined allows designers to explicitly redefine the keywords they are interested in, personalizing the summary to their design priorities.
There has been very limited attention to game review
summarization, besides student projects [
The paper is structured as follows. We start with a review of related works in Section 2. We then describe the proposed summarization pipeline and variants in Section 3. We describe the dataset in Section 4 and present two diferent user studies in Sections 5 and 6. We then discuss the results in Section 7 and conclude the paper in Section 8. 2
User reviews are a rich source of information, although the
extraction and analysis of this information can be challenging not only
due to the textual nature of the medium but also because users tend
to have a mixed opinion about various features [31]. Approaches
such as sentiment analysis as well as summarization have been
applied to various datasets, such as product reviews [24, 31], movie
reviews [
Summarization can be extractive when relevant portions (usually
sentences) of the input are copied and combined, or abstractive
when new text is generated to rephrase and summarize the input
[18]. The summarization pipeline requires a number of steps before
the raw textual input can produce a summary; algorithms and
approaches for each step are discussed below.
2.1.1 Pre-processing and parsing. A fundamental step towards
summarization (and natural language processing more broadly) is the
pre-processing and extraction of features from the dataset. In the
analysis below, the term documents is used to describe any type of
text, e.g. a sentence, a paragraph, or an academic paper. One
popular if naive approach for pre-processing data is the bag-of-words
which collects all words in the document, disregarding their order
and grammar. This method counts the number of instances of the
same word, and the frequency of occurrence of each word is used
as a feature to measure similarity between documents. Since many
words (such as articles or pronouns) are far more frequent in all
documents, terms are weighted based on their frequency via tf.idf
[
Since its 2003 release, the Steam platform has become the largest digital distribution platform for PC gaming [15], hosting over 34,000 games and tens of millions of active users daily. This paper focuses on user-created reviews on Steam, although other initiatives such as the Steam workshop allow users to upload their mods or strategies and comment on others’ content. User reviews can be submitted only by people that have purchased the game from Steam, although they are visible to all. As noted above, Steam aggregates user reviews into a category and provides a number of companion statistics, including a timeline of reviewer’s scores. Reviews themselves consist of a single binary recommendation (Recommended versus Not Recommended) and a text explaining the user’s opinion. Other users can review the quality of the review itself by tagging it helpful, not helpful, funny, or breaking the Rules of Conduct. By default, Steam shows the most helpful reviews submitted within the last 30 days, although users can also choose to sort reviews by other criteria.
As noted in the introduction, there is no systematic academic
research in Steam review summarization. To the best of our
knowledge, the only academic publication that tackles the problem of
aspect-based summarization on such data is by Yauris and
Khodra [
Preprocessing which aims to prepare the input reviews for further analysis. This may imply cleaning, chunking text in snippets or sentences, Part-of-Speech tagging, and other similar tasks.
Aspect Identification which identifies interesting aspects (or topics) in the reviews. These topics may be expressed as a set of words, e.g. "visual, aesthetic, scenery" or "soundscape, audio experience, "sound efects".
Aspect Labeling which assigns clear, descriptive labels to the discovered aspects. E.g. "graphics", "audio".
Sentiment Analysis which gathers information related to the sentiment expressed within the reviews. This information may later be used to update the final summary appropriately. For example, one may need only positive views in the summary, or—most probably—a sampling of all the views, be they positive or negative.
Summary Creation which implies the process which, given all the information gathered in previous steps, forms and renders the final summary for the user.
Given the above pipeline, we implemented three diferent variants. The first two are based on keyword detection and Clustering (CL). The first variant does not do Sentiment Analysis, while the second one uses the full pipeline. The last one is another full pipe method based on Deep Learning (DL) that focuses on improving on Aspect Labeling and Summary Creation steps. 3.1
During the preprocessing step, each review is split into sentences,
each sentence is cleaned in order to create the basic elements on
which the final summaries will be based. The cleaning process
included of some character replacements so that each sentence
could be presentable (e.x. starting with a capital letter and ending
with a period) even if it originated from a larger sentence that was
split during sentence splitting. Moreover, preprocessing prepared
the lemmatized versions of the sentences which are used for aspect
detection. In these lemmatized sentences, general stop words are
removed. For all preprocessing steps, we used the default functions
(and stop word lists) of the nltk Python library [
The aspect detection process is split into two parts: aspect identification and aspect labeling. Aspect identification splits sentences into sets that focus on a specific aspect while aspect labeling identifies this aspect in order to present it to the final review summary.
Our approach uses a predefined set of aspects, presented in Table 1. We selected these six aspects since they are well-established facets of games [28] and are popular dimensions within professional reviews.
A simple approach for aspect labeling is to use a dictionary of
keywords per aspect as the ones presented in Table 1. In order to
be able to include sentences even when they do not include the
exact keywords, a k-means clustering is applied to all sentences to
ifnd clusters with similar text. Terms are weighted based on their
frequency via . , which has been used extensively for sentence
similarity in bag-of-words approaches (see Section 2.1). The result
is clusters of sentences with similar words to each other; in all
our experiments we set = 20 based on prior evidence [
Graphics
Community
Graphics
Gameplay
Gameplay
(1) If the sentence contains the exact keywords of only one aspect, the sentence is assigned to that aspect and is flagged as a candidate that can be used by the summary of that aspect. (2) If keywords from multiple aspects are found in the sentence, the sentence is flagged as an unsuitable candidate for any summary and removed. (3) If no aspect keywords are found in the sentence, the most common aspect within the sentences of the same cluster will be used to label this sentence and flag it as a candidate. For instance, if a sentence does not contain any keyword, but sentences in its cluster predominantly belong to the aspect Gameplay via case (1), then the sentence is also assigned to the same aspect and flagged as a candidate.
Using the sentences from cases (1) and (3), a set of candidate sentences is created per aspect. Using these sets, the first variation of our pipeline could now produce a summary. This variation, named Clustering Aspect Detection summary (CL AsDe), chooses sentences at random from each aspect’s set. A sample CL AsDe summary can be found in Table 2 for Tom Clancy’s The Division.
The next step of the process is Sentiment Analysis, which is used
by the next summarization variant (CL Full). Using the diferent sets
of candidate sentences per aspect, the sentiment polarity (positive or
negative) of each sentence is calculated by averaging the sentiment
score of each word it contains. As above, sentiment analysis of each
word is done via the default functions of the nltk Python library [
After experimenting with the first two variant pipelines and taking into account the feedback of the first user study (see Section 5), we decided to focus on improving the following: • Keyword detection and clustering based Aspect Labeling must be improved to avoid sentences such as "If those things all sound good to you you will like the game." to be labeled as audio sentences. • The final summary should somehow provide information regarding the whole sentiment of the given aspect and not just by the selected sentences. • The final summary should use a better sentence extraction approach in order to deal with redundancy.
Taking all the above into account, the DL pipeline makes changes to the Aspect Detection and Summary Creation steps of the CL pipeline described in Section 3.1.
For Aspect Detection, we used the BERT model [14] to generate
embeddings for game reviews. BERT is a deep neural language
model that uses a bidirectional, multilayer transformer architecture,
exploiting cross and self-attention to capture word
interdependencies efectively [
To obtain the representation for a game review, we feed the text to the model using a sequence length of 16 tokens. We use the model variant, that produces 768-dimensional sequence embeddings, learned during training for classification purposes. The implementation and pretrained model utilized are provided by the transformers software package from huggingface1. Using the produced embeddings as features we trained a binary Ridge Logistic Regression classifier [ 19] (one vs all) for each aspect. We also trained a seventh classifier to detect sentences unfit for any aspect. For each candidate sentence a confidence score was calculated by each aspect classifier. Only sentences with a high prediction confidence in the given aspect and a low confidence on each other classifier were selected as summary candidates for the next steps of the pipeline.
During the Summary Creation we applied the following strategy to the 100 most probable candidate sentences of each aspect. First, the NewSum Toolkit [20] was used to select the sentences that provide the most representative information. NewSum uses language-agnostic methods based on n-gram graphs, that not only extract the most representative sentences, but also deal with redundancy. In the end we had 20 candidate sentences per Aspect. The ifnal summary was composed by 6 sentences using the following strategy: • Select the most positive sentence (Sentiment Analysis). • Select the most negative sentence (Sentiment Analysis). • Select the first 3 sentences provided by NewSum Tookit (excluding the previously selected sentences). • Create an artificial sentence using the polarities provided by Sentiment Analysis of all the aspect sentences. The polarity of each sentence was mapped as 1, 0 or -1 (positive, neutral, negative) using thresholds. Given an Aspect and the mean Polarity score ¯ , the possible produced sentences reflect opinions that fall in the following categories: 1https://huggingface.co/ - In a few words the game is single dimensional this might sound vague but it becomes apparent that there is not much depth as you play once you’re a couple hours in. - Clothes sound "right" when you move in them. - They sound good and looked good with ability to mod for better stats or even rerolling stats. - They have improved the pve portion of the game and crazy as it sounds the pvp too. - No music and something feels so strangely abandnded about it. - Like how if there’s a blizzard your cap and shoulder will be covered in snow and that npc voices will echo when they are standing in hallways with hollow walls. - Very good voice acting. - Great abilities pretty good sounds; indoor echos reverb of objects etc. - If those things all sound good to you you will like the game. - Superb voice acting and ambient city sounds are also a good plus for this game. - It sounds hyperbolic but I’m being dead serious. - Sounds terrible right - Most opinions are positive regarding audio. - The voice acting in the game is in the higher tiers as is most ubisoft games. - There are not a lot of diferent voices and some of the voice acting for them is bad. - Ubisoft - bugs - the textures are so fucked up that nobody can play this game anymore. - And it clearly shows I want to play it and that I try to.
-I’m gonna be honest the cinematics are pretty great.
Table 2: Summaries generated by diferent pipelines, for aspect Audio of Tom Clancy’s The Division. From top to bottom: CL AsDe (only aspect detection), CL Full (aspect detection with sentiment analysis) and DL Full (Deep learning combined with a sophisticated summarizer).
- Mixed: ¯ ≈ 0, high standard deviation. - Mostly neutral: ¯ ≈ 0, low standard deviation. - Mostly positive: ¯ > 0 above a threshold.
- Mostly negative: ¯ < 0 below a threshold.
The final summary is composed by randomly shufling these 6 sentences. An example summary from DL Full variant can be found in Table 2 for Tom Clancy’s The Division.
As a first demonstration of the summarization pipeline, we follow
[
For each of the selected games we selected to keep the 10 thousand most up-voted reviews. As already discussed in Section 3 each of these reviews was split into sentences to create a sentence pool per game. On average, the sentence pool consisted of around 50 thousand sentences per game. The smallest pool of sentences was for PAYDAY 2 (37K), while the largest one was for Elite Dangerous (70K). The average length of the sentences was 85.7 in characters and 16.4 in words. In terms of both characters and words, the longest sentences were those of Darkest Dungeon (average of 91.8 characters and 17.3 words) and the shortest ones were those of Just Survive (average of 79.9 characters and 15.6 words).
In terms of aspects, the most common one was Gameplay on average. Performance was the next most popular aspect and in certain games such as ARK: Survival Evolved it was the most popular one. The least popular aspect was Audio with a ratio of 1 to 5 compared to the Gameplay aspect.
In terms of sentiment, the majority of sentences were more neutral than positive or negative. Between positive and negative sentiment, no general safe conclusions can be drawn since the results varied given diferent combinations of aspects and games. In general, we can say that the aspect Performance was characterized as negative more frequently. The opposite was true for the aspect Graphics. On the other hand the sentiment ratio (positive vs negative) towards the aspect Community varied between diferent games. 5
As a first experiment, we evaluated the two variations of the CL pipeline (CL AsDe and CL Full) in a small-scale user-study with summaries of aspects of the 12 games of Table 3. 5.1
A pairwise comparison process was followed, rather than a
scalebased rating approach, due to (a) evidence that comparison-based
evaluation can be less demanding cognitively [
To this end, we created an online evaluation user interface (UI) (see Figure 2) which supported comparative pairwise evaluation of summaries. We initialized the system by providing two sets of summaries A, B, one from system and one from system . Each summary in A corresponded to a summary in B, as they both summarize the same set of reviews and the same aspect (e.g. the aspect Graphics of DayZ). During the experiment, each system’s summary was randomly placed first or second to minimize any bias related ordering efect.
The UI also informed the user of the title of the game being summarized, plus the aspect (e.g. Graphics). The user was then called to select their preferred summary (A or B) and explain the reasons for this preference. For the latter annotation, the user could select one or more tickboxes among the following options: • It repeats less the same information (Less Redundant) • It seems to be more coherent and/or complete • For other (or even unclear) reasons
The first two options aim to assess whether redundancy is a
concern and, similarly, whether coherence and completeness are useful
in the task. Redundancy has been traditionally a summarization
evaluation indicator [
The evaluation was carried out by eight adult evaluators (3 female), lfuent in English, with gaming experience. The evaluators were selected explicitly among the authors’ network of contacts and invited directly by the authors. Participants were asked to connect to the online system and evaluate all 72 pairs of summaries (produced by CL AsDe and CL Full), which covered all predefined aspects (see Table 1) of the ten games in Table 3. There was no time limit for completing the evaluation, but there was a requirement that all pairs were evaluated in a single session. 5.3
The data collected from the experiment was a total of 576 observations, including the preference of each evaluator for each pair of summaries and the reasons for this choice. The primary goals of the user study are to assess (a) whether the annotators prefer one of the two summarization approaches, (b) which criteria they explicitly (via the three tickboxes) or implicitly (based on properties of the summary) consider when selecting their preference. Towards this end, the data is processed based on the 8 users’ annotations on 72 game/aspect pairs (for a total of 576 data points), and all statistical tests are performed at a 5% significance threshold. Our assumption is that the complete CL pipeline which includes both aspect detection and sentiment analysis will ofer a richer and more diverse summary than AsDe alone.
Regarding users’ preference of one summarization technique, results were mixed: overall, annotators had no clear preference with CL AsDe being marginally more often selected (53%). Table 4 shows the distribution of selection of CL Full split per aspect. The Table shows that the main factor for the skew of the overall preference towards CL AsDe was the graphics summaries, as the other aspects are fairly evenly preferred between the two approaches.
To further assess which factors led to the annotators’ preference of one summary over the other, we conducted an analysis of variance test (ANOVA) between the preferred approach (represented as a binary choice) and other features such as the aspect. Table 5 shows the results in terms of significant diferences, and verifies that there is a systematic influence between the aspect and preference. On the other hand, the game does not seem to afect users’ preference of one summary or the other; this is a promising finding as the methods are supposed to be applicable to any game. There is also a clear evidence that preference was highly varying from annotator to annotator, and annotators rarely agreed with each other even in this simple pair-wise preference task.
To get a better understanding of the reasons annotators gave regarding their preference, we looked further into the statistics of the winning observations of CL AsDe vs. CL Full. When AsDe was preferred, annotators explained their preference mainly due to better coherence (63%), lower redundancy (28%), but also ‘other reasons’ (26%). When CL Full was preferred, annotators chose ‘other reasons’ (50%), and less often coherence (41%) or low redundancy (17%). This finding shows that summaries by AsDe were more coherent but annotators still preferred summaries by CL Full often for other reasons. This points to a limitation of the experimental protocol, as the interface did not provide annotators with enough options to allow them to explain their reasons for their summary preference. This was addressed in the second user study (see Section 6) with an extra option on the UI. It should be noted that better coherence was selected far more often overall (53% of instances) than lower redundancy (23%), while ‘other reasons’ were also chosen often (37%). Redundancy and coherence were chosen together in only 5% of instances, and thus it is evident that these two axes of evaluation are fairly independent. These findings, coupled with the statistically significant influence (via ANOVA) between preference of summarization approach and tagged coherence and redundancy, support our conclusion that both coherence and redundancy were important factors for annotators’ preference. 6
Based on the findings and limitations identified in the first user study, conducted a second study with more participants but fewer games, testing the best CL approaches with the novel DL Full pipeline. Due to participants’ concerns on the long duration of the 72-item survey in the first experiment, we opted to use only two games to lower the time required from annotators; it is expected that fatigue would likely introduce noise to the participants’ responses. Details on how the games and annotation options were chosen are detailed in Section 6.1. 6.1
The user interface for the second user study was largely the same as in the first (see Section 5.1). Based on the first study’s finding that ‘other reasons’ for an annotator’s preference were often chosen, a fourth option was added to the UI as a tickbox stating “The summary was more focused and contained less irrelevant information.” We refer to this additional option as Focus in the analysis that follows.
As noted above, to reduce the time required for the study only two games were chosen to be annotated. We chose among the games from the first user study, taking the game where CL Full had the highest preference (Tom Clancy’s The Division, where CL Full was chosen 60% of the time) and the game where CL AsDe had the highest preference (Elite Dangerous, where CL AsDe was chosen 60% of the time). For each of the two games, the preferred method was chosen to present to the user, juxtaposed with the summary for the same game and aspect produced by DL Full. Therefore, the participant had to annotate 12 items, 6 aspects for Tom Clancy’s the Division comparing the CL Full summary with the DL Full summary and 6 aspects for Elite Dangerous comparing the CL AsDe summary with the DL Full summary. The rationale was to select the most successful game summaries (for both CL variants) and compare them with the novel DL pipeline. We refer to CL and DL summaries in this paper, referring to the best CL summary (CL Full or CL AsDe) as shown to the user.
As with the first user study, the order of the two options was randomized (i.e. sometimes CL summaries were shown first, sometimes second). Unlike the previous experiment, however, the order of the sentences within the same summary was also randomized; the rationale was to avoid ordering efects when the participant starts by reading an incoherent sentence first. 6.2
Fourteen participants completed this annotation task. Unlike the previous study, a snowball method for soliciting participants was followed, soliciting feedback from a broader group. Thus, this study lacks data on the demographics and gaming experience of participants, although participants were all adults and had experience in data analysis and artificial intelligence. 6.3
The data collected from the experiment was a total of 168 observations. Overall CL summaries were slightly more preferred by participants (55%), although the diference is not statistically significant (Paired t-test, p-value 0.22). Interestingly, for Elite Dangerous (which was summarized by CL AsDe) the diference was more pronounced (CL AsDe preferred 60% of the time over DL Full); for Tom Clancy’s The Division the two methods (CL Full and DL Full) were chosen evenly. Since only one game was tested per CL variant, it is dificult to assess whether the preference was due to the game itself or the sentiment-based selection component. Moreover, while DL Full includes sentiment-based selection, this part accounts for 2 of the 6 sentences and thus it is even more dificult to estimate the reasons for the users’ preference. This ambiguity points to further refinements needed for the annotation protocol which is discussed in Section 7.
In terms of the reasons ofered by participants for their choice, coherence was still most commonly chosen (62% of responses), followed closely by focus (56%). Low redundancy was chosen less often (23%), while ‘other reasons’ are chosen only in 14% of responses). The addition of the focus option seems to have mitigated the prevalence of ‘other reasons’ in the first study. Unlike the first study, however, low redundancy was often chosen in conjunction with one other reason (56% of the time) or two other reasons (36% of the time). Combined with its low overall prevalence, it is possible that low redundancy may now longer be necessary as a separate reason in the UI, although a broader user study with more games is needed to validate this hypothesis.
Pearson’s Chi-squared tests were also used in order to test whether any of the above reasons is correlated to the preferred summary. Only redundancy was found to be correlated with the type of summary (p-value 0.001). This clearly indicates the importance of handling redundancy satisfyingly in any future approach. 7
This paper introduced a number of possible pipelines for identifying, grouping, and extracting the opinions of users in terms of pre-specified game facets. two small-scale user surveys examined the preference of users in the presence of diferent pipeline implementations. Results indicate that (a) aspect extraction is important for summarization, although deep-learning does not necessarily improve the aspect extraction process compared to a simpler clustering-based method; (b) between the clustering-based pipeline variants (CL AsDe, CL Full), there was no clear winner with respect to the summary outputs; (c) evaluators had strong and individual opinions on which variant was better; (d) sentiment-based criteria and/or confidence-based criteria for selecting sentences do not seem to perform better than the random selection performed by CL AsDe.
While the aspects chosen for this experiment were intuitive,
based on typical facets of games that players and professional
critics focus on, some of the resulting aspect-based summaries were
less coherent than others. The choice to assign a sentence to an
aspect even if its cluster only had a slim majority in keyword
frequency likely introduced inconsistency. For CL aspect detection,
the most significant factor for the lack of coherence was the choice
of keywords. Specifically, the keyword “sound” was often found in
sentences unrelated to game audio, used as a verb: e.g. “On paper
this game sounds great”. To a degree, such artefacts were removed in
the DL aspect detection pipeline via (a) the latent sentence
representation and (b) fine-tuning the model based on manual annotations
on this specific corpus. However, a more sophisticated method for
aspect detection seems necessary. For instance, an adaptive query
expansion as followed by [29] could create a much larger set of
keywords automatically, although it may overlook the nuances of game
terminology. On the other hand, a Word2Vec model [30] trained
on the entire corpus of steam reviews (or even larger game-related
corpora such as game FAQs and fansites) could be used to derive
a similarity score with specific aspects. Building a game ontology
for this task or using an existing one [
Sentiment analysis was also often problematic, primarily due to
the informal and idiosyncratic language that games reviews were
often in. Reviews are often rife with sarcasm and negation, e.g. “Have
fun spending huge amounts of hours for very little progress.”.
Moreover, many reviews’ sentences have poor syntax and are very short
or very long (e.g. “Good: + great aesthetic.”). Sentiment analysis
treated the sentence as a bag-of-words, exacerbating the problem. In
general, sentiment analysis can not capture negation or sarcasm and
handles incomplete sentences poorly. Performance would likely be
improved with a more appropriate pre-trained lexicon for informal
utterances on the Social Web, such as SentiStrength [
Our findings also showed no clear winner between the two CL variants or between CL and DL summaries. These ambiguity of the ifndings could well be by-products of the experimental protocol followed. Findings from the first user study pointed to a missing reason for players to report, and the second study included a “focus” reason which improved the quality of the data collected but raised questions about the importance of the “low redundancy” reason. The users’ reported fatigue in the first experiment led to fewer items in the second study to alleviate the burden from annotators. However, this increased the locality of the findings in the second study as it was unclear whether preferences were due to the game or the algorithm. In future studies, summaries for more games should be annotated by more participants, showing only two games to each user but randomizing which games are shown when the user starts the study. More importantly, the current experimental protocol forces participants to select one review as preferred and provide at least one reason. The forced choice between two summaries does not allow the user to provide more nuanced feedback. A fouralternative forced-choice (4-AFC) with options “A”, “B”, “both A and B”, “neither A nor B” would allow the user to point out cases where both summaries are equally good or equally bad. The fairly even split between the two alternatives in both user studies could be due to fact that users consider some summaries shown equally bad and select randomly. On the other hand, a 4-AFC questionnaire would likely need many more participants since much of the data will be removed when no ranking is given. The need for more games, more annotation choices, and perhaps more algorithm variants (DL AsDe, for instance) point to the need for a large-scale user survey among the general gaming community, which will be performed in future work in this vein.
As discussed in Section 1 and explored on a high-level during the user study, game review summarization can be valuable both to consumers (players) and producers (game developers). However, each stakeholder has diferent priorities and will likely respond diferently to diferent summary formats. The extractive summarization process was visualized as ‘pure text’ bullet points, which was not as engaging to either type of audience. It would be important to explore alternative visualizations for players and developers. For players, the summary could provide more structure (based on prespecified game facets), focus more on the weights and scoring of each aspect (including visualizations such as pie-charts), show only a few polar opposites in terms of review sentences, and perhaps cross-reference these findings with other games’ review summaries. For developers, on the other hand, a bottom-up topic discovery would likely be beneficial in order to identify unexpected points of contention among users. Moreover, presenting the context of the reviewers’ chosen sentences would also be valuable for designers, e.g. how many reviewers agree with or echo this comment, when this comment was made and whether general sentiment has shifted since then. Such context can be important regarding the urgency of addressing certain concerns or to gauge whether patches and updates have improved reviewers’ perception, not unlike Steam’s use of most recent reviews.
There are many directions for future research depending on the
purpose of the game review summarization. As a tool for game
evaluation, primarily targeted towards players or producers, the game’s
context is important in order to choose which reviews or topics to
highlight. Additional research in this vein would need to find topics
or patterns in similar games (e.g. of the same genre, publisher, or
publication date) and then to compare the current game’s reviews
in terms of those topics or compared to other games’ reviews. User
experience research would also be important to find how best to
present such results, as interactive summaries where the user can
zoom in and out into diferent games and/or diferent topics within
games would make the summaries more intuitive and manageable.
As a tool for game analysis, bottom-up probabilistic topic modelling
[
This paper highlighted the challenges and opportunities of game review summarization via natural language processing. The paper introduced a pipeline for grouping Steam users’ comments into pre-specified aspects such as visuals or performance, and studied diferent renderings of the final summary, exploiting positive and negative sentences based on sentiment analysis. The small-scale user survey revealed diferences in how diferent annotators assess the reviews, highlighted possible foci of research for better game review summarization systems, and suggested a number of refinements to the process are suggested in this promising subfield of game artificial intelligence.