The Copied Item Injection Attack Nathan Oostendorp Rahul Sami School of Information School of Information University of Michigan University of Michigan Ann Arbor, MI 48104 Ann Arbor, MI 48104 oostendo@umich.edu rsami@umich.edu ABSTRACT In many web communities, users are assigned a reputation based For these systems, as with other recommender systems, there is on ratings on their past contributions, and this reputation in turn increasing concern about manipulation by users with a vested influences the recommendation level of their future contributions. interest in promoting or burying certain target items. There is a In this type of system, there is potentially an incentive for authors growing literature on addressing the threat posed by attackers who to copy highly-rated content in order to boost their reputation and create multiple shill or sybil accounts, and then use them to rate influence within the system. We describe this strategy as a items in patterns (perhaps randomized) that will lead to copied-item injection attack. We conduct an empirical study of collaborative filtering algorithms boosting or burying the target this attack on the online news discussion forum Slashdot. We find items. Defense techniques that have been developed include evidence of its use and demonstrate its effectiveness in eliciting detecting and removing anomalous user profiles [2, 17, 12, 20,13], high ratings. We explore variants of this attack in other domains limiting the influence of user profiles until they have made and discuss potential countermeasures.. contributions [19], and providing monetary incentives for honest rating [14, 1]. In this paper, we identify a new class of attacks that user-contributed content recommenders may be vulnerable to: the Categories and Subject Descriptors injection of duplicated or plagiarized items. We study the H.3.3 [Information Search and Retrieval]: Information Filtering prevalence and effectiveness of this attack using a corpus of over 20 million comments from the technology news website Slashdot, General Terms and propose countermeasures against this attack. Reliability, Security. Execution of a copied-item injection attack involves a two-step process for the attacker: First, she must find old items (comments, Keywords on Slashdot) that have been rated very highly by the community. Manipulation, Recommender System, Online Discussion, User- She can then duplicate the entire item, or a portion of the item, Contributed Content and post this to the site as a new item (i.e., a new comment on a different story), claiming to be the creator. Site moderators do not 1. INTRODUCTION always recognize this as a recycled item, and so rate it highly Numerous online communities offer the ability to post and view based on the quality of the original item. In turn, this leads to the user-contributed content, but participants can suffer from reputation of the attacker being increased, as she is the purported information overload in high-traffic environments. Often, rating author of high-quality content. Subsequently, she can exploit this and filtering systems are used to promote content that has been higher reputation, and the improved visibility it brings, to attract created or rated highly by leading users in the community. With attention to subsequent original (and possibly inferior) items she these systems, an item’s initial prominence is often based on the creates. reputation of the content creator. This reputation is based (at least in part) on feedback on items that user has created in the past, and The first question raised by attacks of this form is: Are they serves as a signal of quality as well as an incentive to improve harmful to the site or the rest of the community? This is not quality. For example, reviews by ‘Top Reviewers’ on ePinions obvious, because in some contexts it may be a useful contribution [5] and ‘Elite Members’ on Yelp [22] are prominently displayed, to redirect the community’s attention to valuable information that and comments by Slashdot [21] members with higher ‘Karma’ was known in the past, but has been forgotten. For any given start at a higher level than other comments. domain, this will need to be weighed in comparison to the harm caused by the attack. In section 5, we argue that, for the Slashdot domain, the potential damage caused by this attack outweighs the potential benefit. Existing techniques to prevent or limit manipulation in recommenders do not protect against copied-item injection attacks. This attack does not require the attacker to change her rating profile, so procedures that detect and filter anomalous ratings would not work. The influence-limiting approach also is not effective: Creating a good duplicate, or rating it highly, will be counted as a contribution by the attacker, but in this context, the attacker is merely reusing earlier information from raters on the original item to infer that the copy will be well-liked, but is not actually contributing new information. Existing mechanisms that item i ∈ I has two characteristic features: creator(i) denotes the prescribe monetary incentives to rate honestly do not address this user who is listed as the creator of the item, and content(i) is a problem, as the raters who rate the copies highly are being honest description of its content (text, image features, etc.). The attacker about their perceptions of its quality. In section 6 we discuss some has some target content T that she would like to promote. At any techniques that could be used to combat copied-item injection point in time, an item has a recommendation level rec(i). For attacks. simplicity, we assume that the recommendation level is not personalized; for personalized recommenders, rec(i) could denote The rest of this paper is structured as follows. In section 2, we the average recommendation level among the target community, review the related literature. In section 3, we formalize our or another summary statistic. definition of copied-item injection attacks. Section 4 describes our Each user u has a reputation R(u). Item recommendation level empirical analysis of this attack on the Slashdot dataset, and our rec(i) is assumed to depend on its creator’s current reputation measurements of the current prevalence and effectiveness of this R(creator(i)) as well as the corpus of ratings on the item set I. The attack in the Slashdot domain; we discuss the consequences of user reputation R(u) is assumed to be computed based on the these results in Section 5. In section 6, we discuss corpus of ratings; we assume that, other things being equal, R(u) countermeasures against this threat. We conclude and identify is higher if a particular item i with creator(i)=u has higher directions for future work in section 7. recommendation level rec(i). We assume that two items i,j with content(i)=content(j) have positively correlated recommendation 2. RELATED WORK levels, because the raters cannot consistently identify the later Recently, there has been a rich literature centering on the item as having duplicated content. This is realistic in a system vulnerability of collaborative filtering recommender systems to with a large number of items and users. attack, as well as defenses against those attacks. This was initially observed by Lam and Riedl [8] and O’Mahony et al. [16]. This A copied-item injection attack involves the attacker copying literature has focused on a particular class of threats: attackers can a genuine item i, with a high rec(i), to create a new item c, with create “shill” or “sybil” user profiles, and use these to promote or content(c)=content(i), but creator(c)=a while creator(i)=h. The bury items they have a vested interest in. A number of authors attacker then waits for c to collect a sufficient number of ratings, have studied variants of this attack, as well as defenses against so that rec(c) increases towards the high level of rec(i). Finally, them; we refer readers to recent surveys by Mobasher et al [15] attacker a creates a new item t with creator(t)=a and content(t)=T. and Mehta and Nejdl [13]. Techniques to defend against this The simplest measure of the success of an attack is the attack include methods to detect and remove anomalous user difference between rec(t) after this attack then it would have been profiles [2,17, 12, 20, 13], limiting the influence of user profiles if item c was not created. A slightly more nuanced measure, until they have made contributions [19], and providing monetary which is natural is the context of analyzing a’s incentives, is the incentives for honest rating [14, 1]. The chief difference with our increase in a’s net benefit, accounting for the cost of creating the current work is that we consider a different class of attack: we copy c and the opportunity cost of not creating an original posting study settings in which, in addition to potentially injecting shill instead. We explore this idea further in section 6. user profiles, the attacker can inject items with known quality (derived by copying existing items). 4. ANALYSIS OF SLASHDOT There has also been prior research on the Slashdot moderation Slashdot is a high traffic online news site and an active forum that system. Lampe and Resnick [11] analyze the performance of the receives several thousand user-contributed comments and over a moderation system in identifying high-quality comments, and million pageviews every day [20]. To help the users navigate show that it is largely effective. Lampe and Johnston [9] report among the large amount of user-contributed material, it uses a that new users of the site use the moderation feedback they rating/moderation system that lets them filter comments based on receive as a cue to learn the norms of the community. Lampe et al. a score from -1 to 5. This system has elaborate controls to detect [10] propose to use a second level of collaborative filtering to and discourage abuse, including rules on who can moderate, how adapt users’ interface views of the moderated comments. Poor often they can moderate, and how much they influence the [18] argues that Slashdot is an archetypical public sphere on the score[11]. Internet, and describes the role of the Slashdot moderation system in fulfilling this function. 4.1 Slashdot’s Moderation System David and Pinch [3] conducted a qualitative study of strategic The system revolves around two scores assigned to user accounts: reviewing on Amazon.com. They document several cases of karma, which is accrued by contributing comments and receiving plagiarized reviews; one of the motives they identify for positive moderations on those comments, and mod points, which plagiarizing is to build up a long profile of ratings with low effort. allows users to rate other users comments up or down. In this This is similar to the modus operandi of our copied-item injection system, the users and items are linked by authorship, so that each attack, except that the community’s ratings on the content are item's rating is aggregated into karma for the user. A user’s more important than the raw number of comments in our setting. karma then determines both the probability of acquiring mod points and the starting score for their posted comments. Because 3. MODEL AND TERMINOLOGY positive ratings on an authors comments gives the author In this section, we introduce terminology to clarify our discussion additional influence within the system, there is clear incentive to of the copied-item injection attack. manipulate the system if a users goal is to gain influence or There is a set U of users; we use h to denote an honest prominence in these discussions. contributor, and a to denote the attacker. A set of items I; each Additionally, users can meta-moderate and judge whether a users of 1.158 with a standard deviation of 1.149. 1.30 million mod points have been spent appropriately. In this process, users comments have a rating of 4 or 5, or about 6.2% of the entire can view comment moderation pairs and give a up/down feedback population. on if each moderation was appropriate. Users who frequently are evaluated as having rated inappropriately become less likely to receive mod points. This was designed to defend against simple manipulations where mod points were traded or spent on inferior comments for the express purpose of improving another users karma. While this system has some algorithmic checks for basic profile- injection strategies such as detection of high-traffic cyclical moderation patterns between users, there are some manipulation strategies that can be used to gain undue influence within the system. The online comic WellingtonGrey has humorously documented a few of these in flowchart form [6]. This chart identifies tactics for accruing karma including profile-injection ("a second account with mod points"), strategically expressing popular sentiments in comment text ("Is it about Microsoft? Say they suck. Is it about Apple? Say they rule."). It also advises recycling of old material. ("Do you have any old +5 posts on this topic? Quick, post one!") This third tactic describes copying an item to gain positive ratings, and therefore karma. The Slashdot environment is likely to be an ideal environment for this type of attack, due to several factors. Its longevity as a news source (it celebrated its 10th Anniversary in 2008), and high volume of traffic gives it a large library of existing comments that could be recycled. Since so many comments are posted every day, it is also reasonable to assume readers will be unable to Figure 1: Score Distribution for All Comments on Slashdot recognize an older comment out of the millions authored on the The comment text length distribution is shown in Figure 2 and site. Additionally, the nature of "news cycles" means that certain follows a lognormal distribution. After a logarithmic topics recur frequently: a subject line search shows that Slashdot transformation, the mean comment length is 5.68 (293 characters) has over 200 stories on Windows Vista, which has been in the with a standard deviation of 1.11. The entire body of text from all news for 2-3 years. of these comments is roughly 11.0 billion characters. Based on these factors we can make a few generalizations about where a copied-item attack might be used. Certainly it must have an environment where the cost of item creation is low and also the cost of copying an item is similarly low. The incentive to use the attack must come from when the author receives some indirect benefit from positive ratings on the items they create. The Copied Item attack will also be easier where there are extremely large numbers of items so that the probability of duplication detection by recognition from readers is low. Finally, it will be easier to deploy the attack when items have simple data structures, such as a comment with a block of text, a subject line, and an authorship reference, as opposed to items that might be indexed on many different attributes and therefore may have too many similar attributes to the original. 4.2 Description of Slashdot Data We used a snapshot of Slashdot’s database from January 28, 2009, which contained 20,830,313 comments contributed by 307,158 users across 158,867 news story discussions. Each comment record contained a short subject line, a longer message body, a timestamp of publication, the final rating for the comment, and numerical ids referencing for the story and author. Figure 2: Histogram of Log-Transformed Comment Lengths The rating distribution for comments, shown in Figure 1, is roughly a right-skewed normal distribution centered on the mean 4.3 Detection of Copied Items 4.4 Hypotheses and Results In this study, our goal was to detect plagiarized comments in this Intuitively, we expect that copies of highly-rated comments will large Slashdot comment corpus. The core of this process was also garner high ratings and be useful to potential attackers for the finding comments that shared large substrings. However, there purpose of acquiring karma. In this section we formulate three are several conflating factors which could legitimately lead non- hypotheses that test this conjecture. attackers to reuse large substrings within their comments: users Hypothesis 1: Copying a comment with a high rating is profitable quote from earlier comments or quote the same source; there is a for attackers, in that it produces a comment which is more likely form of political activism that involves posting the same text on average to be highly rated. repeatedly such as the DeCSS decryption codes; and some users attempt to disrupt a forum by posting as many junk comments as If the copying of comments were profitable for an attacker, we possible. We processed the comments conservatively, so that we would expect the copies of these high scoring comments to garner would identify a comment as plagiarized only if none of the higher ratings than the population at large. We found in the target conflating factors is a plausible explanation for the duplicated population of likely plagiarized comments the rating distribution text. of the copied comments was substantially changed versus the distribution of the global population, as illustrated in Figure 3. In order to detect plagiarisms our first step was to detect Indeed population of copied comments had a mean of 2.15 vs the comments that had significant duplicate text. We implemented a global mean of 1.16, nearly a full standard deviation higher than Rabin-Karp search [7] with a window of 255 characters. Using the global mean, a difference of 0.987 points. Additionally, 30.4% this method we converted each 255-character substring of a of items in the copied set had a rating of 4 or 5 as opposed to comment message body into a hash value, and searched for co- 6.2% of the global comment population. A two-sample t-test occurrences of hash values across multiple comments. The entire confirmed significance of both results (p < 0.001). This corpus generated about 6.4 billion (hash,comment_id) pairs. Any discrepancy confirms Hypothesis 1. comment found to have more than 3 hash collisions with any single previously posted comment was logged. We then went through the logged comment pairs and confirmed that there was significant duplicated text using a longest common substring algorithm. This process resulted in 196,349 pairs of potentially plagiarized comments among the 20-million comment corpus. In order to narrow this set of comment pairs to distinguish comments that may have been directly plagiarized with intent to boost ratings, we applied a sequence of filtering steps to the original set of copied items. These included: 1. We removed any pairs where the original comment had a final rating score of 3 or less. This was eliminate comment copies that had little reason to expect a high rating. 2. We removed any pairs where the longest common substring was less than 90% of the copied comment length. This was to avoid comments that had significant original material as well as copied content. 3. We eliminated comment pairs where the copied comment did not begin with the longest common substring. This rule was used to weed out quotations since attributions or quotation marks would typically prefix a quote. 4. We removed any comment pairs that appeared in the same story. This was to avoid implicit quoting within replies. 5. We eliminated comment pairs where the copied comment was posted anonymously, rather than by a logged in user, as Figure 3: Distribution of Scores for Copied Comments anonymous users see no direct benefit from having their Hypothesis 2: Copying a comment with a high rating is more post rated highly. profitable than contribution of other content by the attacker. 6. We eliminated comment pairs where the original comment To see if this strategy is incentive compatible for the attacker, we was copied more than once; this was used to control for looked at our set of copied comments compared with the mean overt reposting, DeCSS code posts, or other forms of rating for the copied item authors other items. By comparing each habitual reposting. of the copied comments scores with the users mean post rating in With these conservative restrictions in place, the set of probable a pair-wise t-test, we found the copied item had a mean plagiarisms was 735 comment pairs, where 423 users had posted improvement of 0.730 points (p < 0.001). This confirms the copied comments. We visually inspected about two dozen Hypothesis 2. pairs manually to confirm that there was no other apparent reason Hypotheses 1 and 2 confirm that copies of highly-rated comments for duplication. tend to be rated highly even when taken out of their original context. It is conceivable that these comments add value to the readers of multiple topics, and that little damage is done by hash encoding of the entire comment text. Subsequent comments rewarding the copiers for reposting them. We will discuss the that were posted with the same MD5 sum as a previous comment harm caused by the copied-item attack in more detail in section were rejected from the discussion. We looked at our copied 5.1. Here, we provide evidence that the copies damage the comments sample set and found 28 comments posted before this signaling quality of the Slashdot rating system: feature was deployed, 26 of which were exact copies. After this Hypothesis 3: The average rating of comments, other than the change it was not possible to post the identical comment again; copied comment, by the copier is lower than the average rating of however, it was possible to make a trivial change to a comment, other comments by the original poster. such as addition of whitespace, and repost. Of the 707 copies detected dated after the March 20, 2001, 618 were identical to the In order to test this hypothesis, we first excluded all instances in original except for the insertion or deletion of punctuation and/or which the original comment was posted by an anonymous user. (If whitespace. After controlling for whitespace and non- the original comment was posted by an anonymous user, we could alphanumeric characters we found no significant difference not identify other comments posted by the same user; further, it is between entire/partial match ratio between the two populations clear to the readers that a comment is anonymous, and hence it is using a binomial test. unlikely that they would improve their expectation of other anonymous comments). For each of the 683 surviving instances, We suspect that this may possibly be due to the extreme ease with we measured the average rating of all comments (other than the which a duplicated post could be altered by adding even a single copied comment) posted by the original poster, and the average whitespace character anywhere in the text. It also may be that our rating of all comments (other than the copied comment) posted by conservative heuristics used to detect likely plagiarisms select the copier. We find that the average rating for the original poster primarily towards exact matches in this data set. is 1.70, vs 1.38 for the copiers; a two-sample t-test confirms significance (p <0.001). This suggests that the copiers actually had lower quality than the original posters, and thus, the high 5.1 Is Slashdot comment copying really rating they receive for the copied content reduces the ability of harmful behavior? readers to distinguish them from the higher-quality posters who From a certain perspective, it may be reasonable to point out that posted the original comments. the copied comments on Slashdot do add value to the system. In a Hypothesis 4: Copied comments are much more likely to be topic sense, the positive ratings that the duplicated comments receive starters (comments starting a discussion thread) than other are signals from the raters that the comment has value, and this comments, since it would be more difficult to have a copied may add insights that otherwise wouldn’t be seen in this response seem appropriate as a reply to multiple comments. discussion environment. While it may take a certain moral flexibility to ignore the taboo of plagiarism, the copied item We looked at the location of our copied comment population in posters could be thought of as agents of conversational arbitrage, Slashdot discussions and found that of the 734 copied comments seeking out and shining up old gems from previous discussions. 573 were topic starters. If you contrast this with the entire However, simply looking at the reposted comments as harmless comment population of 20.8 million, 6.28 million comments injections ignores other externalities of having unattributed started topics. A two-sample t-test indicates that the copied reposting in a discussion system. Although the user ratings reflect comments are 47.8% (78.0% vs 30.2%) more likely to be topic the immediate visceral reaction of the raters to the content, this starting than a comment in general (p < 0.001). Hypothesis 3 is may not capture the entire value of a piece of content to the therefore confirmed. The consequence of this hypothesis is that system. copying can distort the pattern of interaction on the site, skewing it towards breadth rather than depth of exchange. For the Slashdot domain, we believe that the potential damage outweighs the potential benefit: Users can always jog the 5. DISCUSSION community’s memory by quoting earlier comments with attribution instead of resorting to plagiarizing comments, and With H1, H2, and H3 confirmed, it seems evident that item quoting is fairly widespread, so there is little additional benefit copying has been successfully used on Slashdot to systematically accrued through these attacks. In fact, given the availability of garner high ratings for comments and therefore improve the users quoting as an alternative which meets community norms and karma score. We expect that this type of item injection attack has requires negligible additional effort by the copier, the fact that a potential to be a widespread problem both in the realm of Slashdot user would choose to not credit the original author is illuminating: and other moderation-based comment systems as well as other it indicates that they expect to gain a better reception (and better collaborative filtering spaces. In any forum where inserting ratings) by suppressing the fact that the content was duplicated. copies of highly rated content is incentive compatible and This in itself suggests that the ratings are not perfectly aligned technically possible there is a strong likelihood of abuse. At the with the community’s perception of the long-term value of a core of this incentive problem on Slashdot is the transitive contribution. property of item scores to users, where a user stands to directly gain influence in the system by receiving positive feedback on their items. However, systems containing low-cost item creation It is likely any systemic method to gain karma would have an may present different incentives for this type of attack, and it may undesirable effect on the Slashdot system, and become create variations in overall impact. increasingly widespread if the technique was communicated between users. One problem is this tactic distorts karma as a Simple manipulations to try and disrupt this type of behavior may signal of someone who has contributed good fresh content. For add only marginal costs to the effort required to copy comments. instance, in the Slashdot system, karma has a direct impact on the On March 20, 2001 Slashdot deployed a code update that starting score of a users post. Therefore a user with high karma attempted to curtail comment “re-posting” by logging an MD5 user may start their post at 2, rather than 0 or 1. This means that the comment ratings lose their effectiveness as a signal of quality Another application in which copying items can increase the as well in this particular situation. This loss of signaling quality power of an attacker is in search engine website rankings. Here, was borne out in our confirmation of hypothesis 3. the `ratings’ are expressed in the form of other sites linking to a The other potential impact if this tactic of copying comments was particular site. By copying some content from a high-quality site, widespread is that it would have a negative impact on the dynamic an unscrupulous site operator can increase the chances of other actual conversations that occur within Slashdot. Hypothesis 4 genuine sites linking to his site. This will drive up the ranking of confirms that these comments tend to be discussion-topic starters, his site on search engine results pages; some of these pages can be but any replies to these copied comments would be very likely to used to damage readers through unrelated advertisements or be disregarded by the attacker. They are, after all talking to a fraudulent content. different person than the user who originally generated the There are several other domains that could potentially see item- comment text. This means in as copied comments became more injection attacks. In the news website space, gaming a frequent within the system, the harder it would be for users to find collaboratively filtered news aggregator such as Digg [3] could be genuinely interactive experiences. profitable by increasing traffic and therefore ad revenue. Ultimately, we believe the threat is significant enough that defenses against it merit careful consideration. This phenomenon 6. POTENTIAL COUNTERMEASURES potentially weakens both incentive and signaling function of the In this section, we describe a framework for reasoning about site’s reputation system: Users may be incentivized to copy items countermeasures to the copied-item injection attack, and identify as a lower-cost way of building reputation than creating original several possible techniques that could effectively combat this content, even though the latter is a more valuable contribution; threat. There are two core factors behind the copied-item attack: and, future original contributions by the attacker may start as a (1) Users have an incentive to increase their reputation, and incur misleadingly high recommendation level, because they reflect the effort costs when they attempt to do so, either by copying items or quality of the author of the original item, rather than the attacker’s by creating fresh contributions. (2) Copied items are likely to inherent quality. Additionally, it may create an incentive for garner ratings that are similar to those of the original item. We copying content without attributing the original author, which can frame our discussion of countermeasures with these two aspects disrupt the norms of the online community. of the problem in mind. For a given domain, it is helpful to visualize a space A of 5.2 Variants in other domains possible pieces of content, coupled with a distance metric that It is possible that a copied item injection attack could potentially captures the similarity between two pieces of content: The smaller appear in other types of recommender spaces where items can be the distance between x and y, the more similar the pieces of inserted into the system with relatively minor barriers, just as content. For example, A could be the space of all text strings, and profile injection attacks are potentially problematic in spaces the distance measure could be based on edit distance, or keyword where a user creation in a system is extremely low-cost. In frequencies. For a movie domain, A could be defined by a set of particular, any systems where ratings on items transitively score features (title, actors, director, etc.), with a distance metric based the users who create the items will provide incentive for this type on this feature similarity. Modeling the content space in this way of attack. allows us to reason about near-copies as well as exact copies. The Although the Slashdot recommender system uses a simple voting cost and benefit to an attacker a in executing an item-copy method of collaborative filtering, it is sophisticated in tracking injection attack can then be described in terms of this reference reputations for users and using these reputations to allocate model. When a copies an item i to generate a near-copy item c, visibility and influence. Reputation tracking is a powerful method her cost is presumably increasing in the distance between of identifying high-quality contributors over time, so we expect content(i) and content(c), reflecting the effort of obfuscating the that many recommenders for social web applications will adopt fact that the item was copied; for example, it takes some effort to some it in some form. Then, copied-item injection attacks, reword a comment or change the whitespace and punctuation. The perhaps in conjunction with other attacks, will become a potential benefit accruing to the attacker depends on the ratings that c threat. garners; given that i was a very highly-rated item, the benefit In particular, it is the combination of an item and profile attack might be highest for an exact copy but drop off as the distance between content(i) and content(c) increases. that could be extremely problematic. A sophisticated attacker could use the copied items to establish validity for shill items Techniques to combat item-copy injection attacks can work posted by shill accounts, and likewise rate other comments by raising the cost of carrying out the attack, imposing a penalty if similarly with shill accounts. This would potentially create a the attack is detected, or reducing the benefit of creating the copy system where scores could be quickly increased on both shill item c. users and items. • One natural technique is to detect copies, and either In a movie recommender system (or other traditional item prohibit them outright, or impose a reputation penalty recommenders) a combination of an item and user injection could when they are injected. This is the approach that Slashdot potentially distort recommender predictions if site maintainers implemented when they prohibited exact copies of were not vigilant about repairing duplication. A copied item, comments. In practice, however, this imposes an whether legitimately cataloged as a variant of an original film (ie insignificant cost on attackers, as they only have to make a “directors cut”) or sorted under a different name, could be used trivial changes to a previous comment. Instead of merely as a target item in a manipulative attack in order to “push” or identifying exact copies, a slightly more sophisticated “nuke” according to an agents agenda. approach might detect an item within a certain distance of a pre-existing piece of content, using a distance metric moderators could be shown nearest content items, and appropriate for the domain. This has a two-fold might be more skilled at distinguishing genuine forms of advantage: it forces attackers to put in more effort in copying from reputation-boosting plagiarism. The modifying the original content, and in doing so, the copy tradeoff, of course, is that this requires additional human is less similar to the original item, leading to a lower effort that might be better spent in creating or rating expected benefit. Another variation would be to not items. In addition, as with rating systems, there would prohibit near copies, but rather, to merge similar items need to be a system to prevent attacker shills from into a single logical ‘item-cluster’. controlling this moderation process, perhaps necessitating a level of “meta-moderation” as well. There are two drawbacks to this approach, however. First, it is only as good as the distance metric used. This might One constraint on all of these techniques is that calculating spark an arms race between attackers and site managers, distances between pieces of content in a large database can be in which attackers continually find clever ways to retain very computationally intensive. This might preclude the use of the quality of the original item while appearing to be these techniques in a online mode. Instead, the automated distant under the current metric, and site managers techniques could be used offline to periodically filter items or continuously update the metrics to plug these gaps. adjust reputations. Human moderators trying to locate similar Second, as the distance threshold increases, there is a pieces of content online would have to rely on simple distance growing threat of false positives: genuine items that get metrics. mistaken for copies. This could hamper the contribution of honest users. It is not possible to meaningfully evaluate the performance of these techniques on our existing dataset, as the attackers are likely • Alternatively, the defense can focus on reducing the to adapt the detailed form of attack once a specific benefit to users of copying items, relative to more socially countermeasure has been deployed. This is borne out by the way valuable activities such as the creation of original content. in which users sidestepped Slashdot’s check for identical The attacker derives benefit because of the increase in her comments, as described in section 5. The evaluation of the relative reputation and the privileges that accompany a better effectiveness of these countermeasures is therefore left as a reputation. This suggests that a more sophisticated subject for future work. reputation update may be effective: When a user a creates an item i, rather than increase her reputation based merely on the average rating of i, we should account for the average rating of similar items as well. For example, the 7. FUTURE WORK creator’s contribution might be calculated as the In this paper, we have identified a class of attacks, copied-item difference between the average rating of item i and the injection attacks, that user-generated content recommenders on average rating of the nearest (in terms of content distance) the web may be vulnerable to. We have studied this attack in a pre-existing item j; or, perhaps, use a similarity-weighted single domain, but the attack pattern is relevant to many different average of all pre-existing items. This reduces the benefit settings; likewise, countermeasures developed in one setting will of copying high-quality items, hopefully to the point that be helpful in others as well. There are several important directions users choose more valuable ways of building their for future work. The development and implementation of practical reputation. Genuine posting of similar items would still countermeasures should be a priority for applications where the be possible, but there would be a reduced incentive to do copied item injection attack is a feasible strategy. For some so. domains where duplicate detection of content is impractical, one direction of research may be to use patterns of user ratings to The same approach can be extended to tailor the identify similarity between items. incentives of raters as well as creators. The Influence Additionally, it would be useful to conduct empirical or Limiter [18] scores raters based on the amount they experimental measurement of the prevalence of this attack in improve predictions for future raters. Loosely, a rater who other domains. This would give confirmation as well as a broader is the first to rate a high-quality item high will gain the understanding of attack patterns and the motivations of attackers. highest score, while subsequent raters will be measured as having diminishing contributions. A rater’s accumulated Once countermeasures have been implemented and deployed, and score is then used to limit their influence on others’ users have had a chance to adapt to them, it will be important to predictions. In the case of a profile injection attack, the experimentally determine their effectiveness by comparing the effectiveness of each shill is stunted – as it adds no frequency and impact of attacks with and without defenses. information, it will not earn a high reputation score, and hence have limited influence. As described in [18], the 8. ACKNOWLEDGMENTS Influence Limiter might be susceptible to copied-item This work was supported by the National Science Foundation injection attacks: The attacker expects the copy c to have under award IIS-0812042. We would also like to thank the similar ratings to the original i, and thus, attacker shills Slashdot Engineering team at SourceForge Inc, specifically Rob can be the first to put in high ratings where relevant. This Malda, Jamie McCarthy, and Uriah Welcome for their help in can be countered by scoring the early raters on items accessing and interpreting Slashdot comment data. We are also relative to a benchmark prediction that is the average of grateful to Paul Resnick at the University of Michigan for his pre-existing items with similar content. helpful feedback and suggestions on this project. • A third technique might be to rely on targeted moderation that flags items as ‘legitimate’ or ‘plagiarized’. Human 9. REFERENCES [12] B. Mehta, T. Hoffman, and P. Fankhauser. Lies and [1] R. Bhattacharjee and A. Goel. Algorithms and incentives for propaganda:detecting spam users in collaborative filtering. In robust ranking. In Proceedings of the ACM-SIAM Proceedings of IUI’07, 2007. Symposium on Discrete Algorithms (SODA ’07), 2007. [13] B. Mehta and W. Nejdl. Attack resistant collaborative [2] P.-A. Chirita, W. Nejdl, and C. Zamfir. Preventing shilling filtering. In Proceedings of ACM SIGIR ‘08, 2008. attacks in online recommender systems. In WIDM 05, pages [14] N. Miller, P. Resnick, and R. Zeckhauser. Eliciting honest 67–74, 2005. feedback: The peer-prediction method. Management Science, [3] S. David and T. Pinch. Six degrees of reputation: The use 51(9):1359–1373, 2005. and abuse of online review and recommendation systems. [15] B. Mobasher, R. Burke, R. Bhaumik, and C. Williams. First Monday, 6, 2006. Towards trustworthy recommender systems: An analysis of [4] Digg, 2009. http://www.digg.com. attack models and algorithm robustness. ACM Transactions on Internet Technology, 7(2):1–40, 2007. [5] Epinions, 2009. http://www.epinions.com. [16] M. O’Mahony, N. Hurley, and G. Silvestre. Promoting [6] W. Grey. The slashdot flowchart, 2007. recommendations: An attack on collaborative filtering. In http://miscellanea.wellingtongrey.net/2007/04/28/slashdotflo Proceedings of the 13th International Conference on wchart/ Database and Expert System Applications, pages 494–503. [7] R. M. Karp and M. Rabin. Efficient randomized pattern- Springer-Verlag, 2002. matching algorithms. IBM J. Res. Dev., 31(2):249–260, [17] M. P. O’Mahony, N. J. Hurley, and G. C. M. Silvestre. 1987. Detecting noise in recommender system databases. In [8] S. K. Lam and J. Riedl. Shilling recommender systems for Proceedings of the 2006 International Conference on fun and profit. In Proceedings of WWW ’04., pages 393– Intelligent User Interfaces, pages 109–115, 2006. 402, 2004. [18] N. Poor. Mechanisms of an online public sphere: The [9] C. Lampe and E. Johnston. Follow the (slash) dot: effects of website slashdot. Journal of Computer-Mediated feedback on new members in an online community. In Communication, 10(2), 2005. Proceedings of the 2005 international ACM SIGGROUP [19] P. Resnick and R. Sami. The influence limiter: Provably conference on supporting group work, 2005. manipulation-resistant recommender systems. In Proceedings [10] C. Lampe, E. Johnston, and P. Resnick. Follow the reader: of the ACM Recommender Systems Conference (RecSys07), Filtering comments on slashdot. In Proceedings of CHI 07 2007. Conference on Human Factors in Computing Systems, pages [20] J. Sandvig, B. Mobasher, and R. Burke. Robustness of 1253–1262, 2007. collaborative recommendation based on association rule [11] C. Lampe and P. Resnick. Slash(dot) and burn: Distributed mining. In Proceedings of the 2007 ACM Conference on moderation in a large online conversation space. In Recommender Systems, 2007. Proceedings of ACM CHI 2004 Conference on Human [21] Slashdot, 2009. http://www.slashdot.com. Factors in Computing Systems, 2004. [22] Yelp, 2009. http://www.yelp.com.