=Paper=
{{Paper
|id=Vol-2480/GHItaly19_paper_01
|storemode=property
|title=Analysis of Advertising in E-Sports Broadcasts
|pdfUrl=https://ceur-ws.org/Vol-2480/GHItaly19_paper_01.pdf
|volume=Vol-2480
|authors=Kalle Kareinen,Antti Knutas,Jussi Kasurinen
|dblpUrl=https://dblp.org/rec/conf/chitaly/KareinenKK19
}}
==Analysis of Advertising in E-Sports Broadcasts==
https://ceur-ws.org/Vol-2480/GHItaly19_paper_01.pdf
Analysis of Advertising in E-Sports Broadcasts
Kalle Kareinen Antti Knutas Jussi Kasurinen
LUT University LUT University LUT University
Lappeenranta, Finland Lappeenranta, Finland Lappeenranta, Finland
antti.knutas@lut.fi jussi.kasurinen@lut.fi
ABSTRACT the same time, streaming services such as Twitch have given
During this decade, esports has become popular, both as an opportunities for anyone to find an audience, just by
industry and as a spectator sport. However, advertising in the broadcasting themselves playing video games. In fact, online
esports broadcasts has not yet reached its full potential if gaming leagues such as ESEA and FACEIT have provided
compared to areas such as TV or radio. The goal for this opportunities for individual players to show their skills to the
paper was to develop a new kind of advertisement tool using bigger audiences and to the gaming organizations while
game data to have more interesting and visible advertisement reaching for the goal of becoming professional gamers [3,4].
compared to the traditional advertising methods. The
implementation was made for the broadcasts of the video In the video game Counter-Strike: Global Offensive
game Counter-Strike: Global Offensive, and its effectiveness (CSGO) two teams of five players try to reach 16 round wins
was measured by a survey. The developed implementation to achieve an overall map win [5]. Rounds can last from 15-
runs automatically with no actions needed by the 20 seconds up to two and a half minutes, and it is possible
broadcaster, and according to the survey, the implementation that all the players are involved in the action at the same time.
was liked by the respondents, but its advertisement was not At times, the action might be chaotic and even the
remembered any better than traditional advertisements. experienced observers and commentators might have
difficulties in noticing all the key moments. In the beginning
Author Keywords of every round in the game, there is also a so-called
Esports; advertisements; Broadcasts “freezetime”, which is usually the time for the commentators
ACM Classification Keywords to discuss the actions of the previous round.
• Software and its engineering~Software implementation By developing a new analyzing tool for commentators using
planning the Game State Integration of CSGO, it is possible to help
INTRODUCTION the commentators with their analysis and to provide a new
Esports broadcasts are still a minor player compared to advertising element to the broadcasts. Helping the
traditional sports broadcasts seen in television and radio. commentators with their analysis helps to create more
However, esports have the possibility to develop new attractive broadcasts to gain more viewership both to the
advertising methods compared to the traditional sports broadcasting channel, and to the teams that are playing.
broadcasts, since they the potential to apply more adaptive When the statistics are shown for the audience, it provides
and sophisticated technologies. Even still, static both new information to the viewers and at the same time
advertisements which have been seen in traditional sports allows advertising the sponsors of the broadcast. After all,
arenas for decades, are usually the only approach even if the advertisement providing information is less irritating for the
effectiveness of the static advertisements of the traditional viewer than a non-informational advertisement [6].
sports has been questioned even before the first major esports The main goal for this paper was to implement a new
tournaments [1]. Since video games provide new advertising tool which is also a useful analyzing tool.
possibilities to how advertisements are presented, there is an Implementation will be used for analysis to define the key
incentive to study how they could be utilized. moments of the round for the viewers and allow for
During the last five years, the money involved in the esports informative and esthetic user interface (UI), while allowing
industry has grown to hundreds of millions of dollars [2]. At a sponsor logo to be seen at the same time. The UI element
of the implementation will be more of a Proof of Concept
instead of a real, production-ready element, with logos
placed in the space that is left unused by the presentation of
statistics. Other goals are to document the implementation
and measure the effectiveness of the advertisements against
the traditional static advertisements and logo areas.
To achieve this, Design Science Research (DSR) was used
GHItaly19: 3rd Workshop on Games-Human Interaction, September 23rd, 2019, in this paper by developing a new artifact to help to
Padova (Italy) understand the chaotic rounds of CSGO and to provide a new
Copyright © 2019 for this paper by its authors. Use permitted under Creative advertising method at the same time. The research process
Commons License Attribution 4.0 International (CC BY 4.0).
consisted of identifying a problem before designing and spots for the advertisements since there are no gameplay
developing a new artifact to help with the problem and elements in the area to be obstructed (e.g. [12]). The common
finally evaluating the artifact [7]. Overall, the research placement of advertisements is demonstrated in Figure 1.
question is “How to design a more noticeable advertisement
In a study by Grace et al [13], it was studied if in-game
for esports broadcasts without making audience more
advertising was more effective for onlookers than players,
irritated by it?”
which are familiar with the electronic entertainment. The
The rest of the paper is structured as follows. Section 2 research concluded that viewers who have played the game
discusses the works related to our research, whereas Sections themselves can be more motivated to focus on the core game
3 and 4 discuss the setting and the obtained results. Section mechanics instead of watching the advertisements. In the
5 discusses the potential implications, and Section 6 closes research it was found that onlookers remembered the in-
the paper with the conclusions. game billboard advertisements significantly better than the
players who are more familiar with the game, mainly because
RELATED RESEARCH
The in-game advertisements have emerged as a new platform onlookers are less motivated to look at the game mechanics,
for companies to advertise their products in esports and shifting their focus more towards the advertisements [13].
digital entertainment. There are also a lot of different
methods that can be used to raise awareness of the brands
that are sponsoring game developers to select from. For
example, Ip [8] lists main types of in-game advertising being
for example real life or movie characters as in-game
characters or branded in-game assets or tools.
There are also entire game franchises based on movie
franchises, such as the Lego video games for both PC and
gaming consoles [9], or sports games using person brands or
purchased licenses to use real life teams and players,
including logos and sponsors [10]. The study by Ip noted that
this type of in-game advertising is sometimes not even
considered as advertising by the players, because it adds Figure 1: Typical advertisement locations
immersion to the product. The lack of interaction with the RESEARCH SETTING
advertisements decreases the likelihood of the player As an additional advertising model, some tournament
noticing the advertisement and the advertised brand with it organizers have used branded replays showing the logo and
[8]. However, it is noted that it is difficult to measure how name of the advertising brand while playing the replays of
much these advertisements have subtle effects on player’s key moments of the rounds [14]. Forming the replays and
perceptions of the advertised brands. showing them on screen is difficult if the broadcaster does
Research paper by Li et al. [11] studied a possibility of not have at least one dedicated person to parse the replay, or
adding advertisement automatically to the broadcast. The if the system does not offer one automatically.
research listed several reasons why virtual advertisements, Implementation of our tool informs the viewer about the key
advertisements inserted into the user interface layout, are moments of the rounds without any actions by the
more effective than static billboard advertisements and broadcaster. The implementation gathers this data
commercial breaks. For example, billboards might not be automatically, builds the presentation with the advertised
seen because other objects could be in front of them, or that logos, and slides it to the broadcast without any actions from
users simply do not watch the broadcast during a commercial the broadcaster.
break. With virtual advertisements, the user has to watch also The implemented ad system is shown to the audience when
the ads if they wish to see the content. The criteria for placing the current round is over, and hidden when the next round
advertisements is also discussed, and it includes elements starts. Implementation provides both information about the
such as that the advertisements should not block any ongoing match and a changing position for the advertised
foreground objects because that annoys the audience, or that logos. It also automatically applies the Game State
advertisement should stay static for a period of time so that Integration (GSI) of CSGO which enables the possibility to
it can attract the attention of the audience. fetch useful in-game data which cannot be seen in the
Esports broadcasts have an advantage with virtual graphical user interface of the game.
advertisements because in terms of gameplay footage, the Using the GSI requires a listening server, which catches
environment and camera setups are virtual too, unlike in the network requests made by the game. The data provided in the
traditional sports broadcasts. Also, when the gameplay is requests needs to be analyzed by a program to find out if it is
shown in the broadcast, it is always known which areas of valuable or not. After analyzing the data, it needs to be
the screen are less informative, meaning that they are good presented with a simple yet attractive way to catch the eye of
Phases
which some examples are presented in Figure 3. Imaginary
organizations were used to simplify the permission process,
4
and to eliminate the popularity or recognizability bias
between the different brands.
SURVEY SETUP
1
Evaluation of the implementation was measured by creating
Kills 3 two surveys. The surveys were almost identical, but Group
A did not see the end-round advertisements at all, only the
classic static advertisements of the UI. The comparison
Group B had the same static advertisements, but they could
2
see also the implementation advertisements in the summary
screen. With this arrangement it was possible to find out if
Figure 2: Implemented advertisement tool ad locations implementation advertisements had any effect, and compare
them to the static advertisements. Technically this approach
the viewer towards both the provided statistics and the logo was a type of A/B testing, where the focus was on the added
of the sponsor. This backend was made with the Node.js features, not on the directly competing solutions [16]. Open
server which picks up HTML POST requests from the game feedback was also collected as it contained more structured
client, while the frontend is a web page which can be viewed opinions from the respondents. In the surveys, there were
in a browser or in the streaming software. Node.js server three different methods to measure the usefulness of the
receives JavaScript Object Notation (JSON) formatted implementation:
payloads from the game, analyzes them and sends parsed
data to the frontend using a web socket made by socket.io 1. How many advertisements respondents could remember
library. 2. Measuring opinions about advertising method(s)
3. Open questions (voluntary for respondents)
Web page updates itself depending on the data sent by
backend and is served by the Express web application Mann-Whitney U test was used to find out if there were
framework [15]. This implementation does not use any kind significant differences between the opinions of A and B
of database, the information is stored only in the web browser group about the static advertisements. It was also used to see
and the runtime of the backend during the time it is open on if respondents of group B had significant differences of
a client system. When the round is over, the implementation opinion regarding the advertisement printed in the
tries to calculate the best possible placement for the implementation compared to the static advertisements.
advertised logo. There will always be room for the All the respondents were given a link to a Google Form
advertisement in either the top-right corner or the bottom-left which contained a link to an unlisted YouTube video and
corner because of the UI design: If there is not enough room survey questions. Both groups saw a video of the same round
for the advertisement in the bottom left corner, all the of CSGO match with the same game audio and voice
eliminations of the round have occurred in the beginning of commentary. Both surveys were done fully online. Both
the round leaving the top-right corner empty and if there is videos lasted two minutes and 44 seconds and contained four
not enough room for the advertisement in the top-right static advertisements. Group A saw four static
corner, eliminations and actions of the round have occurred advertisements in the center bottom of the screen while
in the later seconds of the round leaving bottom-left corner Group B saw the same static advertisements and the
empty. Following this logic, the advertisement will be placed implemented summary screen advertisements. Static logos
in the center of the leftover space between the drawn were visible the whole time, two minutes and 44 seconds in
elements, and the borders of the canvas. The order in which both videos, while the implementation was shown only 17
the placement of the advertisement will be attempted is 1) seconds during the round ending time and “freezetime”. The
Middle-left, 2) Bottom-left, 3) Middle-right and last 4) Top- original round was recorded using Open Broadcaster
right. Figure 2 presents a simplified demonstration how the Software (OBS) Studio which was also used to inject the
implementation tries to place the advertised logos, from advertisements. It was decided not to talk about
advertisements in the commentary of the survey so that both
groups could have the same commentary. The survey was
done by respondents with the following process: 1) Watch
the video, 2) Close the video before answering how many
advertised brands or logos could be memorized, and 3)
Possibility open the video again and answer questions about
static advertisements. Group B was also asked to answer
Figure 3: Three examples of the made-up logos: Swift mobile questionnaire concerning the summary as a fourth phase.
was applied only in the summary screen, while others were
used in the static advertisements.
Questions of the survey were modified from Viewer Group B saw also the additional summary screen
Response Profile (VRP) items [6]. VRP questions are advertisement. Basic comparison of only mean and median
designed originally for television commercials and because values of observations yielded only little information; the
of that, questions were modified for the needs of static Group A and Group B answered similarly to the degree
advertisements and the implementation. The purpose of the where there were no statistically significant differences.
VRP items was to test three things: 1) Evaluate whether there However, if we observe the Group B and the different
were significant differences in opinions towards static advertisement styles, there are differences illustrated in
advertisements if the more active implementation Table 6.
advertisement was shown, 2) Evaluate if respondents
Table 6. Comparison of static vs. implemented summary
enjoyed the implementation advertisement significantly
screen advertisement types with Group B.
more than the static advertisements, and 3) Evaluate if
respondents thought the implemented advertisement was too Question U p Mean Mean Med. Med.
(static) (impl.) (stat) (impl.
complex. Mann-Whitney U method was used to find out if )
there were significant differences in two different sets of I thought the advertising 109.5 .00024 2.46 3.88 2 4
responses. method was clever and
entertaining.
According to conventions statistical hypothesis testing, the These are the kind of 114.5 .00036 1.58 2.88 1 3
hypothesis is confirmed if p value is less than 0.05 [17]. advertisements that keep
running through your mind
Basically, the null hypothesis h0 is defined as following for after you have seen them.
every question h0: There is no significant difference between The advertisement was too 195 .05614 1.58 2.08 1 2
advertising methods and h1: There is a significant difference complex. I was not sure what
between advertising methods. was going on.
It required a lot of effort to 258 .54186 2.58 2.25 2 2
SURVEY RESULTS follow the advertisements.
Listing the advertised brands from memory went poorly for These kinds of advertisements 76 <.0000 4 2.21 4 2
nearly all the respondents. Some of the respondents in group have been done many times. It 1
is the same old thing.
A thought that the team names were the only advertised I have seen this advertising 139 .00222 2.92 1.75 3 1.5
companies in the video. Even when the advertisements were method so many times—I am
noticed, only a maximum of 1-2 advertised brands were tired of it.
remembered and more than half of the respondents could not I think that this is an unusual 75.5 <.0000 1.5 3.17 1 3.5
advertising method, I am not 1
name any of the companies. In the group A, 13 respondents sure I’ve seen another one like
out of 23 did not remember any companies at all. The it.
numbers were similar in group B, where 15 respondents out The advertising method 213 .12356 2.08 1.42 1.5 1
of 24 could not remember anything particular. Only one irritated me — it was
annoying
participant listed all information perfectly, as an outlier.
Many responses contained partly correct names, so
evaluation was done by using a point system where correct For example, “I thought the advertising method was clever
answer was 1 and partly correct answer 0.5p. Unfortunately and entertaining.” had a quite neutral value from the Group
for the implemented summary screen and our test control, B for the static advertisements, 2.46, while the implemented
Swift Mobile was not the most remembered company. The advertisements got much more positive feedback, score 3.88.
ads in the summary screen implementation were noticed, but The next question, “These are the kind of advertisements that
the advertisement in it was not remembered any better than keep running through your mind after you have seen them.”
two of the four static advertisements. In this sense, the had similar difference, 1.58 compared to 2.88 so
implemented tool was not successful. The results for implementation was received better in both questions of this
remembered companies are summarized in Table 5. category.
Table 5. Points per brand in the listing task. Confusion category had two questions and neither of them
had significant difference between the advertisement types.
Brand Group A Group A Group B Group B However, “The advertisement was too complex. I was not
points % points %
Swift Mobile - - 3 12.5%
sure what was going on.” was pretty close to statistical
Ultralight 4.5 19.6% 3 12.5% significance (p = .05614) with static advertisements having
Networks unsurprisingly good mean of 1.58 and implementation
TropicBets 3 13.0% 4 16.7% having 2.08. The difference was expected in this question
Buzz 2 8.7% 1 4.2% because there was not really anything to understand with the
Strongtech 1 4.3% 0 0%
static advertisements while the implementation included
Overall, there were 23 respondents in the group A and 24 statistics without explaining them beforehand. The second
respondents in the group B, collected mainly from volunteers question, “It required a lot of effort to follow the
in a number of college-student aimed esports and gaming advertisements.” was very neutral for both advertisement
events. Group A had only the static advertisements, whereas
types since the static advertisements had mean of 2.58 and Two respondents pointed out the biggest problems of both
the implementation had 2.25. the static commercials and the survey rather well:
All three questions for the familiarity category had “Problem with static commercials is that they can't be where
significant differences as expected. The first two questions one actually keeps their eyes while watching CS:GO. My
had slightly negative tone for the advertisement being too eyes never went down where the commercials were the whole
familiar. The static advertisements received 4.00 mean for time.[…]” -Respondent #1 Group B
the question “These kinds of advertisements have been done
“The best thing about them is that they don't cover up any of
many times. It is the same old thing.” While the
the vital gameplay elements, and that they aren't distracting.
implementation received 2.21 for the same question. “I have
I feel that they would be more effective when you're exposed
seen this advertising method so many times—I am tired of
to them for a longer period of time.” -Respondent #17 Group
it.” was tonally more aggressive and it received 2.92 for the
B
static advertisements, and only 1.75 for the implementation
since implementation has not been used in many broadcasts Respondent #16 noted importantly, that the static
yet and included a surprise factor of round statistics. The commercials are usually known brands instead of made-up
third question “I think that this is an unusual advertising companies, which helps viewers to spot the logos:
method, I am not sure I’ve seen another one like it.” had a
“The advertised companies should be related to gameplay so
mean of only 1.5 for the static advertisements while the
that people who usually watch the stream can relate to them
implementation had 3.17. In the first and third question p
better ie. HyperX, Nvidia etc. ...”- Respondent #16
value was smaller than .00001 which means very significant
difference when significant difference is smaller than 0.05. There were some complaints that advertisements should
The only question in the alienation category was important have had bigger logos and texts to be easier to recognize:
for the research question, because it measures the irritation “I don't even pay attention to them in the first place. I had to
towards the advertisements: “The advertising method rewatch the video to answer the first question. Also they are
irritated me — it was annoying”. There was no significant pretty small.” -Respondent #5 Group A
difference between the advertising methods in this question
with p being .12356. As a positive aspect for to the “Too many logos that were all too small in my opinion” -
implementation, it had better mean of 1.42 against 2.08 of Respondent #11 Group B
the static advertisements. Overall the results of Mann- DISCUSSION
Whitney U indicate that summary screen and its The implemented advertisement system was designed for the
advertisements were a significantly more entertaining and needs of esports broadcasting. There were no strict
newer advertising method which was not seen as more requirements other than offering extra information and a spot
irritating by the viewers. Additionally, the result for the for a new advertisement at the same time. The survey did not
question “These are the kind of advertisements that keep measure opinions on the information given by the
running through your mind after you have seen them.” was implementation, but the open feedback provided some ideas
also significantly better for the implementation and pointed out problems which could be fixed. In terms of
advertisement. Design Science Research, a new artifact was created to help
with existing problem, and it was evaluated using a survey.
Additionally, open feedback was collected from all of the
The research cycle and the basic guidelines of Design
participant groups. The open feedback provided valuable
Science Research was adapted from Guideline by Hevner
information about the advertisements, survey and on the
[7]. In general, the validity of the research was evaluated by
aspect of what the viewers actually notice in esports
using Wohlin et al. [17] as a guideline.
broadcasts in general. Even though the summary screen
implementation was seen as an enjoyable advertising method The implementation was more of a “proof of concept”
by the respondents, there were some constructive feedback instead of being an advertising tool ready to be used in an
about separating the advertisement and statistics more official tournament broadcast. In the future, there should
clearly: probably be a presentation text with the logo, for example:
“Round recap provided by”, which would help new viewers
“The design of the advert could be improved so that it is
to understand more quickly which part of the implementation
clear that the round summary and the advert are two
are placed advertisements and which are statistics and
separate entities.” -Respondent #7 Group B
gameplay information.
Both the static advertisements and the implementation could
The video clip in the survey lasted only one round while the
have looked a little bit more polished. As the UI of the
viewers spend usually a lot more time watching the stream.
implementation was more of a “Proof of Concept”, the
A regular match lasts at least 16 rounds so there is a lot more
following feedback did not come as a surprise:
time for the viewers to look at the advertisements during the
“Needs to have some pretty graphics so it doesn't look so broadcast. A more in-depth research on this subject would
rough for the viewers.” - Respondent #10 Group B include watching a longer video with more polished
advertisements. It would probably help if the advertised 4. ”ESEA”. [Online]. Available: https://play.esea.net/
companies were real, existing brands and were not [Accessed: 29-Nov-2018].
completely new and unknown to the users, like in this test 5. ”About CS:GO”. [Online]. Available:
run. The weak results of the question “List all advertisements http://blog.counter-strike.net/index.php/about/
you can remember” were indicative of the users missing the [Accessed: 28-Nov-2018].
advertisement spots, but on the other hand the results show
how easily the advertised brands are ignored. Overall, by 6. A. Strasheim, L. Pitt & A. Caruana, ”Psychometric
using imaginary logos we also sidestepped the problem of Properties of the Schlinger Viewer Response Profile
the brand recognition being different for different user (VRP): Evidence from a Large Sample”, Journal of
groups. Advertising, 36:4, 2007, p101-114.
7. Hevner & Chatterjee, ”Design Research in Information
If a similar study is done in the future, there could also be a
Systems - Theory and Practice”, Springer Integrated
method which was used by Grace et al. [13] where addition
Series in Information Systems 22, 2010, pp. 12,16.
to listing all the advertisements from memory, all the eight
logos and 16 extra logos were shown to the respondents. In 8. B. Ip, ”Product placement in interactive games”, ACM,
future studies, this could improve the overall accuracy now ACE ’09, Proceedings of the International Conference
that we have a baseline in which no extra logos were applied. on Advances in Computer Enterntainment Technology,
2009, pp. 89-97.
CONCLUSIONS
Advertising methods in esports should be examined more 9. ”Video Games”. [Online]. Available:
thoroughly because the industry is still growing. The esports https://www.lego.com/en-gb/games/videogames
industry and broadcasts are full of advertising opportunities [Accessed: 27-May-2018].
different from the traditional sports broadcasts but the static 10. ”Behind the scenes, worldwide licensing team works
advertisements are still used to a surprising degree. In this on FIFA soccer video game”. [Online]. Available:
paper, we provided a solution to deliver extra information http://www.theprovince.com/technology/behind+scene
automatically, and studied the static advertisements against s+worldwide+licensing+team+works+fifa+soccer+vide
more adaptive approach, which based on our prototype o+game/9007347/story.html [Accessed: 2-Dec-2018].
results was considered more appropriate method of
11. Y. Li, K. Wah Wan, Y. Yan & C. Xu, ”Real Time
advertising in an esport event.
Advertisement Insertion in Baseball Video Based on
It was found that this survey did not deliver strong results, Advertisement Effect”.
but considering the open feedback given by respondents, 12. ”Astralis vs. North Highlights | Grand Final |
implementation advertisement is a good addition to the CORSAIR DreamHack Masters Stockholm 2018”.
esports broadcast advertising methods. It provides useful [YouTube video]. Available:
information about the previous round, instead of being just https://www.youtube.com/watch?v=90ocyPYG_xI
another advertisement. It was noted that the looks of the [Accessed: 27-May-2018].
implementation should be refined and that too excessive use
of it as an advertising method could make it irritative 13. L. Grace, D. P. Janssen & J. R. Coyle, ”Did You See
similarly to the static advertisements. That? In-Game Advertising Retention in Players and
Onlookers”, ACM, ACE ’14, November 11-14, 2014.
The research question “How to design a more noticeable
14. ”Luminosity vs Vitality - Overpass - DreamHack Open
advertisement for esports broadcasts without making
Atlanta 2018” . [Youtube video]. Available:
audience more irritated by it?” was not completely solved,
https://youtu.be/jAzcShMDGRs?t=774 [Accessed: 01-
but steps were taken towards the right direction. The overall
Dec-2018].
result might have been different with real logos or by making
the logo more visual. However, all the three objectives for 15. ”Express - Node.js web application framework”.
the research were completed, and the implementation yielded [Online]. Available: https://expressjs.com/ [Accessed:
useful information for future designs and refinement. 01-Dec-2018].
REFERENCES 16. P. Hynninen & M. Kauppinen, ”A/B Testing: A
1. L.W. Turley & J. R. Shannon, ”The impact and Promising Tool for Customer Value Evaluation”,
effectiveness of advertisements in a sports arena”, IEEE, 2014.
Journal of Services Marketing, vol. 14 No. 4 2000, pp. 17. C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B.
323-336. Regnell & A. Wesslén, ”Experimentation in Software
2. ”Esport earnings”. [Online]. Available: Engineering”, Springer (2012), pp. 68-69, 102-104..
https://www.esportsearnings.com/ [Accessed: 7-May-
2018].
3. ”FACEIT”. [Online]. Available:
https://www.faceit.com/en [Accessed: 29-Nov-2018].