Astrotur ng is the practice of using a fake online social media (OSM) pro le in order to in uence public opinion, while giving the impression that the pro le belongs to an authentic human user. In attempting to train a classi er for discriminating between authentic users and astroturfers, a labeled dataset must rst be arranged. The labeling is generally done manually, by human judges, on a collection of pro les garnered from the social media network. However, the fact that any randomly collected set of pro les will statistically contain a small proportion of astroturfers, renders this process ine cient: a lot of time and e ort is invested on manually labeling lots of data, while producing only a small set of astroturfer pro les. We present here a method for quickly and efciently collecting a data set for manual labeling, with a high percent of astroturfers.
Along with the growing use of social networks, so has the realization of its
dangers grown, albeit arguably at a slower pace. Adversarial use of online social
media (OSM) pro les comes forth in various scenarios, such as social ("Cyber
Bullying"), nancial, health (disseminating anti-vax pseudo scienti c claims).
We focus here on the political scenario. The practice of attempting to create a
false pretense of wide public support of a speci c candidate in the political eld,
by using a fake OSM pro le, is known as astrotur ng. [
One approach for solving this problem is to harvest suspicious accounts by
using what are known as \social honeypots" [
Our dataset consists of close to half a million Facebook pro les collected over a span of 15 months, from close to four million comments on some twenty political candidates' posts. During those 15 months, the political system in Israel went through one upheaval after another, with three rounds of elections attempting to reach a decree that would enable the formation of a government. This rare situation created for us a rich dataset, with many novel attributes useful for identifying astroturfers. 2
Utilizing the Innate Honeypot Nature of Political Pro les As mentioned, creating arti cial honeypots in order to attract political astroturfers would not work. However, we came to realize that the existing pro les of the political candidates are in fact natural honeypots, in that they attract, by their very nature, the astroturfer pro les. This in itself is not enough, since authentic users are also attracted to these honeypots, and as mentioned, the ratio of labeled astroturfers to authentic users who interact with political pro les, is very low. This is where one of the many features we studied presented itself as useful for the task at hand. We noticed that astroturfers are quick to pounce on fresh posts, rendering the pool of the rst several commentors rich with astroturfers. The method we suggest consists of collecting pro les which are among the rst several commentors on the posts, and presenting these pro les to the human judges for labeling. We posited that the percent of astroturfers labeled would be signi cantly higher than for a random collection of pro les. We present here data to support this thesis.
We described above the motivation for this study: having only 2.25% of the training set labeled as astroturfers, which is not much to show for the huge amount of e ort which went into the task. Our rst attempt was based on the realization that the proportion of comments astroturfers produce is much higher than their own proportion in the population. Therefore, instead of sampling the pro les, we sampled the comments, and chose those pro les which produced the sampled comments. This approach indeed brought forth a higher proportion of astroturfers labeled|46 of 400 pro les (11.5%). This is better, but not enough.
Our next attempt was to apply a preliminary manual sifting of pro les, creating two sets: one of suspected pro les and one of innocent looking ones. These two sets were then presented to the judges for labeling, using the same guidelines. The preliminary sifting proved useful|of 364 pro les, 76 (20.88%) were labeled as astroturfers. This is much better, however it required extra e ort expended for the preliminary sifting.
However, the signi cant growth in the proportion of labeled astroturfers was achieved by what we present here as our method: instead of choosing from all comments, we chose only from the top 10 comments. Apparently, the tendency to comment as quickly as possible is driven by astroturfers' high motivation to disseminate and promote their agenda immediately, whenever the opportunity presents itself, in the form of a new post. This insight reveals the innate honeypot nature of political pro les|there is no need to create fake pro les in order to lure the astroturfers. These pro les already exist, they just need to be utilized in the right way. We tested this hypothesis by targeting those pro les responsible for the rst 10 comments on 50 randomly chosen posts. The pro les were targeted by randomly sampling with returns from the pool of 500 comments (top 10 from 50 posts). It is worth noting that these 500 comments belonged to some 300 commentors|a fact which can be attributed to the nature of such quick-tocomment users, who also have a tendency to comment more than once. Labeling this pool of pro les uncovered 25% of them as astroturfers.
Figure 1 summarizes and compares all four methods|Random Baseline (where pro les were selected randomly from among all pro les), Chosen Posts (where pro les were selected by choosing randomly from comments rather than from pro les), Past Selection (after preliminary sifting, as done in our past research) and Top 10 (which is the proposed method of choosing from among the rst 10 comments). Clearly the Top 10 has the highest ratio of labeled astroturfers. Not far behind is the Past Selection, however recall that the manual labour invested in this result was far greater, so that the ratio of pro les produced to e ort expended is even more pronounced than what the gure shows. These are only preliminary results. The number of rst comments we targeted in this study was 10, however this is no magic number. Further studies should compare results for di erent values of K, using the rst K comments. It is important to stress again that the human annotators received training with detailed criteria and guidelines before they began labeling the data. These guidelines did not change, throughout the various pro le sets presented for labeling. In addition, it should be kept in mind that this method is pertinent only to the rst stage of preparing training data for the classi cation algorithms. The correct choice of algorithms and parameters can then be found, applied and analyzed, in order to create a high performance classi er. A large set of labeled data is a critical resource, and a method for achieving such a set without wasting precious time and manual e ort is valuable. We intend to make this valuable repository of validated astroturfer pro les available for the use of the research community.