Sentiment analysis has become a popular and challenging area of research in computational linguistics (e.g., [3, 6]) and even digital humanities (e.g., [10]), encompassing a range of research activities. Sentiment is often more complicated than a positive/neutral/negative distinction, dealing with a wider range of emotions (cf. [2]), and it can be applied to a range of types of text, e.g., on YouTube comments [9]. Sentiment is but one aspect of meaning, however, and in some situations it can be difficult to speak of sentiment without referencing other semantic properties. We focus on developing an annotation scheme for YouTube comments, tying together comment relevance, sentiment, and, in our case, humor.
Our overall project goal is to develop techniques to automatically determine which of two videos is deemed funnier by the collective users of YouTube. There is work on automatically categorizing YouTube videos on the basis of their comments [5] and on automatically analyzing humor [4]. Our setting is novel in that for YouTube comments each comment does not necessarily itself contain anything humorous, but rather points to the humor within another source, namely its associated video (bearing some commonality with text commentary analysis, e.g., [11]).
For our annotation of user comments on YouTube humor videos, a standard binary (+/-) funny annotation would ignore many complexities, stemming from different user motivations to leave comments, none of which include explicitly answering our question. We often find comments such as Thumbs up for Reginald D. Hunter! (https://www.youtube.com/watch?v=tAwZL3n9kGE), which is clearly positive, but it is unclear whether it is about funniness.
We have developed a multi-level annotation scheme (section 3) for a range of video types (section 2) and have annotated user comments for a pilot set of videos. We have also investigated the impact of annotator differences on automatic classification (section 5). A second contribution of this work, then, is to investigate the connection between annotator variation and machine learning outcomes, an important step in the annotation cycle [8] and in comparing annotation schemes.
Our first attempt to extract a set of funny videos via the categories assigned by the video uploader failed, as many videos labeled as comedy were mislabeled (e.g., a comedy video of a train passing through a station). Thus, we started a collection with a diversity of different categories, covering different types of humor, and began gathering this data semi-automatically. To ensure broad coverage of different varieties, we formed a seed set of 20 videos by asking for videos from family, friends, and peers. We asked for videos that: a) they found hilarious; b) someone said was hilarious but was not to them (or vice versa); and c) “love it or hate it” types of videos.1 The seed set covers videos belonging to the categories: parody, stand-up, homemade, sketch, and prank. We used the Google YouTube API (https://developers.google.com/youtube/) to obtain 20 related videos for each seed (˜100 total videos), filtering by the YouTube comedy tag. The API sorts comments by the time they were posted, and we collected the 100 most recent comments for each video (˜10,000 total comments). In case of conversations (indicated by the Google+ activity ID), only the first comment was retrieved. Non-English comments were filtered via simple language identification heuristics. 3
Intuitively, annotation is simple: For every comment, an annotation should mark
whether the user thinks a video is funny or not—perhaps allowing for neutral or
undetermined cases. But consider, e.g., a sketch comedy video concerning Scottish
accents (http://www.youtube.com/watch?v=BncDeMO_en0). While there are
obvious cases for annotation like “LOL" and “these shit is funny”, there are cases
such as in (
(
In (1a) the user is expressing a positive comment, which—while related to the video—does not express any attitude or information concerning the sketch itself. In (1b), on the other hand, the comment is about the video contents, informing other users that the clip comes from a show Burnistoun and that the user finds this show to be fantastic. Two problems arise in classifying this comment: 1) it is about the general show, not this particular clip, and 2) it expresses positive sentiment, but not directly about humor. Example (1c) shows the user quoting the clip, and, as such, there again may be an inference that they have positive feelings about the video, possibly about its humor. The degree of inference that an annotator should 1There will be some bias in this method of collection, as humor is subjective and culturally specific; by starting from a diverse set of authors, we hope this is mitigated to some extent. Level Relevance Sentiment Humor
Labels
R(elevant)
P(ositive) N(egative) Q(uote) A(dd-on) U(nclear) F(unny) N(ot Funny) U(nclear) F N U I(rr.) U(nsure) draw during the annotation process must be spelled out in the annotation scheme in order to obtain consistency in the annotations. For example, do we annotate a comment as being about funniness only when this is explicitly stated? Or do we use this category when it is reasonably clear that the comment implies that a clip is funny? Where do we draw the boundaries?
Funniness is thus not the only relevant dimension to examine, even when our ultimate goal is to categorize comments based on funniness. We account for this by employing a tripartite annotation scheme, with each level dependent upon the previous level; this is summarized with individual components of a tag in table 1. The details are discussed below, though one can see here that certain tags are only applicable if a previous layer of annotation indicates it, e.g., the F funniness tag only applies if there is sentiment (Positive or Negative) present. For examples of the categories and of difficult cases, see section 4.
This scheme was developed in an iterative process, based on discussion and on disagreements when annotating comments from a small set of videos. Each annotator was instructed to watch the video, annotate based on the current guidelines, and discuss difficult cases at a weekly meeting. We have piloted annotation on approx. 20 videos, with six videos used for a machine learning pilot (section 5). 3.1
First, we look at relevance, asking annotators to consider whether the comment is
relevant to the contents of the video, as opposed to side topics on other aspects
of the video such as the cinematography, lighting, music, setting, or general topic
of the video (e.g., homeschooling). Example (
(
Turning to the labels themselves, annotators choose to tag the comment as R
(relevant), I (irrelevant), or U (unsure). As mentioned, since relevance is based on
the content of the video, comments about the topic of the video are not
considered relevant. Thus, the comment in (
Okay, but homeschooling is not that bad!
Relevance—and particularly relevance to the video’s humor—is a complicated
concept [1]. For one thing, comments about an actor’s performance are
generally deemed relevant to the video, as people’s impressions of actors are often tied
to their subjective impression of a video’s content and humor. In a similar vein
regarding sentiment, general reactions to the contents of the video are also
considered relevant, even if they do not directly discuss the contents of the video. Several
examples are shown in (
(
One other notion of relevancy is concerned with the idea of interpretability and
whether another user will be able to interpret the comment as relevant. For
example, all non-English comments are marked as irrelevant, regardless of the
content conveyed in the language they are written in. Likewise, if the user makes a
comment that either the annotator does not understand or thinks no one else will
understand, the comment is deemed irrelevant. For example, a video of someone’s
upside-down forehead (bearing a resemblance to a face) generates the comment
in (
(
Sentiment measures whether the comment expresses a positive or negative opinion, regardless of the opinion target. Based on the assumption that quotes from the video make up a special case with unclear sentiment status (cf. (1c)), annotators choose from: P (positive), N (negative), Q (quote), A (add-on), and U (unclear). Q is used for direct quotes from the video (without any additions), and U is used for cases where there is sentiment but it is unclear whether it is positive or negative.
For example, in (
A is used in cases where the comment responds to something said or done in the
video, usually by attempting to add a joke by referencing something in the video.
An example comment tagged as A is shown in (
The positive and negative cases are the ones we are most interested in, for
example, as in the clear case of positive sentiment in (
(
In terms of both relevance and sentiment, we use a quasi-Gricean idea: If an
annotator can make a reasonable inference about relevance or sentiment, they should
mark the video as such. In (
(
With the comment expressing some sentiment, annotators then mark whether the
comment mentions the funniness of the video (F) or not (N)—or if it is unclear (U).
In this case, we are stricter about the definition: if it is not clearly about funniness,
it should not be marked as such. For example, the different comments in (
(
Note that by the time we get to the third and final layer of annotation, many preliminary questions of uncertainty have been handled, allowing annotators to focus only on the question of whether the user is commenting on the video’s humor.
We present examples for cases that can easily be handled by an annotation scheme and others that are more difficult. The latter indicate where guidelines need refinement and where automatic systems may encounter difficulties.
Clear Cases Consider the comment in (11a), annotated as RPF: The comment
directly mentions the video (R), has positive sentiment (P), and directly comments
on the humor of the video, as indicated by the word “hilarious" (F). The comment
in (11b), in contrast, makes it clear that the viewer did not find the video to be funny
at all, garnering an RNF tag: a relevant (R) negative (N) comment about funniness
(F). Perhaps a bit trickier, the comment in (11c) is RNN, being relevant (R) and
expressive of a negative opinion (N), but not commenting on the funniness of the
video (N). While sometimes it is challenging to sort out general from humorous
sentiment, here the general negative opinion is obvious but the humor is not.
(
c. I DO NOT LIKE HOW THAY DID THAT
Turning to comments which do not use all levels of annotation: The comment in (12a) is a quote from the video and is tagged as being relevant and a quote: RQ. Finally, there are clear irrelevant cases, requiring only one level of annotation. The comment in (12b), for example, is tagged as I: It is not about the content of the video, and this annotation will not move on.
(12) a. MCOOOOYYY!!! b. Subscribe to my channel Difficult Cases Other comments prove to be more difficult to make judgments. One concept that underwent several iterations of fine-tuning was that of relevance. As discussed in section 3.1, certain aspects of a video are irrelevant, though determining which ones are or are not can be debatable.
Consider the discussion of actors, specifically as to whether a comment refers to the specific role in the video or the overall quality of their work. In (13a), for example, the comment is about the comedian as a person, not relevant to his performance in the video (I). We can see this more clearly in the distinction between the comment in (13b), which is Irrelevant, and the one in (13c), which is Relevant. (13) a. Almost reminds me of Jim Carry ! :) lol she looks just like him Girl version b. Kevin hart is always awesome ! c. I love kevin hart in this video d. The name of this video is Ironic considering its coming from Cracked ;) e. Homeschooling sucks worst mistake I ever made and I lost almost all social contact until college. It’s great academically but terrible socially From a different perspective, consider the video’s title, e.g., in (13d): As we consider the title a part of the content of the video (in this case displaying an ironic situation), the comment is considered relevant. The emoticon suggests a positive emotion, and there is no reference to funniness (RPN).
Perhaps the most challenging conceptual issue with relevance is to distinguish the topic of the video from the contents. The comment in (13e) seems strongly negative, but, while it discusses the topic of the video (a comedy sketch on homeschooling), the opinion concerns the topic more generally, not the video itself: I.
Moving beyond relevance, other comments present challenges in teasing apart the opinion towards the video’s contents versus opinions about other matters. The comment in (14a), for example, is relevant because it is about the video’s content. This backhanded compliment must be counted as positive because, while insulting to all women, the user is complimenting the woman in the video. Thus, the comment is annotated RPF.
(14) a. Very funny for a woman b. 104 people are [f***ing] dropped!
The comment in (14b) has a less direct interpretation, outside of the YouTube commenting context, in that it refers to the thumbs up/down counts for the video. While the comment does not directly address the content of the video, it indirectly does by referencing fraternity habits from the video (as in “dropped from pledging a fraternity”). The comment is annotated as positive despite the negative tone expressed because it implies that the 104 people who downvoted the video are wrong. Consequently, the comment is labeled RPN.
Once relevance and sentiment have been determined, there are still issues in terms of whether the comment is about funniness. In (15a), for instance, the relevant comment is clearly positive, but its status as being about funniness is unclear, necessitating the label RPU. Likewise, the comment in (15b) is negative with unclear funniness (RNU).
(15) a. Dude this go a three-pete! awesome! b. Such a dated reference c. Smiled...never laughed
From a different perspective, the comment in (15c) conveys a certain ambivalence, both about the sentiment (positive, but not overwhelmingly so) and about the funniness, distinguishing either between different kinds of humor or between humor/funniness and something else (i.e., something that induces smiling but not laughing). In such cases, we annotate it as RPU, showing that the uncertain label helps deal with the gray area between clearly about funniness and clearly not.2
The comments in this section are cases that could only be annotated after an intense discussion of the annotation scheme and a clarification of the annotation guidelines. Additionally, the comments show that annotating for funniness or sentiment based on sentiment-bearing words is often not sufficient and can be misleading; (13e) is an example of this, as an irrelevant comment is filled with negative sentiment words. We need to consider the underlying intention of the comment, often expressed only implicitly. While this means an automatic approach to classifying the comments will be extremely challenging, such types of cases show that our current scheme is robust enough to handle these difficulties. 5
We have observed that, despite intensive exposure to the annotation scheme, our annotators make different decisions.3 Consequently, we decided to investigate whether the differences in the annotations between annotators have an influence on automatic classification. If such decisions have an influence on the automatic annotations, we may need to adapt our guidelines further to increase agreement. If the differences between annotators do not have (much of) an effect on the automatic learner, we may be able to continue with the current annotations.
Since the goal is to investigate the influence of individual annotator decisions, we perform a tightly controlled experiment, using six videos4 and four different annotators. To gauge the variance amongst different annotation styles, we performed machine learning experiments within each video and annotator. The task of the machine learner is to determine the complex label resulting from the concatenation of the three levels of annotation. Thus, we have four separate experiments, one for each annotator, and we compare results across those. This means the task is relatively easy because the training comments originate from the same video as the test comments, and out-of-vocabulary words are less of an issue, as well as video-specific funniness indicators being present (e.g., mentions of “twelve” in the Homeschooling video). But the task is also difficult because of the extremely small size of the training data given the fine granularity of the target categories. For each video and annotator, we perform threefold cross-validation.
We use Gradient Boosting Decision Trees as a classifier, as implemented in Scikit-learn [7] (http://scikit-learn.org/stable/) to predict the tripartite labels. We conduct experiments using default settings with no parameter optimiza2We only provide a single annotation for each comment, giving the most specific funniness level appropriate for any sentence in the comment; while future work could utilize comments on a sentential or clausal level, we did not encounter many problematic cases.
3Space precludes a full discussion of inter-annotator agreement (IAA); depending upon how one averages IAA scores across four annotators, one finds Level 1 agreement of 83% and agreement for all three levels of annotation around 61–62%.
4IDs: V97lvUKYisA, Q9UDVyUzJ1g, zvLpXIYLrec, cFkIJBVZ4_w, WmIS_icNcLk, rLw-9dpHtcU Video Tig Notaro J. Phoenix Water Homeschooling Kevin Hart Spider tion. This setting is intended to keep the parameters stable across all videos to allow better comparability. We use bag-of-words features (recording word presence/absence) since they usually establish a good baseline for machine learning. Results Table 2 shows the results for the experiments using default settings for the classifier. Comparing across videos, Homeschooling has the highest average machine learner performance while Tig Notaro and Kevin Hart have the lowest.5
Comparing across videos, the results show that all videos seem to present a similar level of difficulty, with Homeschooling being the easiest video and Kevin Hart the most difficult, based on averaged classification results. However, accuracies vary considerably between annotators. For example, A1’s annotations for Tig Notaro resulted in dramatically lower ML accuracies than A2’s. For Spider, the opposite is true. The differences between the highest and lowest result per video can be as high as 15% (absolute), for the Spider video. These results make it clear that the different choices of annotators have a considerable effect on the accuracy of the machine learner. 6
In this paper, we have presented a tripartite annotation scheme for annotating comments about the funniness of YouTube videos. We have shown that humor is a complex concept and that it is necessary to annotate it in the context of relevance to the video and of sentiment. Our investigations show that differences in annotation have a considerable influence on the quality of automatic annotations via a machine learner. This means that reaching consistent annotations between annotators is of extreme importance.
5If one compares the average results to IAA rates, there is no clear correlation, indicating that IAA is not the only factor determining classifier accuracy.