In this paper, we present the Webis Mastodon Corpus 2024, a collection of about 733 million public posts from the timelines of 1,015 Mastodon nodes across 61 days. Mastodon is a federated open-source microblogging platform that gained a lot of attention in 2023 as an alternative to Twitter (now rebranded as X). However, searching Mastodon is not straightforward due to its federated architecture. This presents an interesting new challenge for federated IR research, and our corpus is meant as a starting point for the new direction of federated microblog search. To ensure privacy, we host the corpus on TIREx, where it can be processed but neither read nor downloaded, with the goal of developing a shared task and a public leaderboard. We also publish our parallelized and polite Mastodon crawler alongside this paper.1 In the wake of Twitter's self-inflicted demise, several new competing microblogging services have emerged. They took the chance to grow by inviting the users that wanted to leave Twitter onto their platforms. Many of these users divided across three platforms: Mastodon, Bluesky, and Threads.2,3 Mastodon is unique among them in that it implements the federated open-source social networking protocol ActivityPub [1], a W3C standard, which forms the basis of the Fediverse, the federated “universe” of social networks that can communicate with each other via this protocol.4 The Fediverse in general, and Mastodon in particular, are proclaimed to be less vulnerable to platform decay [2] than proprietary platforms. Their core principles, namely openness, federation, and independence of the attention economics, give users a lot of agency, from creating Mastodon nodes with custom rules to options for controlling content visibility. Mastodon is highly interesting for researchers, because, similar to Wikipedia, the inner mechanisms of a large social media platform are publicly visible, and because promising research results can be more directly transferred to practice. This is also true for search and retrieval on Mastodon, which presents new and unique challenges due to the federation of the platform. In September 2023, a consensual, node-level search functionality has been integrated into the platform, which enables search on a given Mastodon node but not across
nodes. Searching the federated network of Mastodon nodes as a whole, however, is still dificult. A federated search would require the commitment and reliability of all participating nodes, and to balance efectiveness with eficiency while respecting user and node preferences for visibility and consent.
In this paper, we create the foundation for research on federated search on Mastodon by creating the Webis Mastodon Corpus 2024 (Section 3) and by analyzing it with respect to the perspectives and limitations for search on Mastodon (Section 4). Our collection consists of about 733 million posts from the local and federated timelines of 1,015 diverse instances, spanning 61 days worth of Mastodon trafic (see Table 1). We ofer limited, privacy-preserving access to this collection by hosting it on TIREx [ 4], The Information Retrieval Experiment Platform, which is implemented on top of TIRA [5], the TIRA Integrated Research Architecture. We already invite researchers to contribute retrieval systems for a future shared task on federated microblog search on Mastodon.
Mastodon has attracted a fair amount of scholarly attention since its inception in 2017, but not comparable in volume to research on Twitter, Facebook, or Reddit. Existing work is interested in social relations and structures formed in the federated scenario [6, 7, 8], in content moderation [9] and governance [10], and, very recently, in migration patterns towards the Fediverse [11, 12, 13].
While there is some prior work in information retrieval on Mastodon (e.g., account recommendation [14]), most work on search is driven by community initiatives like propulsion.social to search nodes. However, none of these initiatives search for posts, and those that attempted to do so, like search.noc.social and fedsearch.io, were shut down citing "extreme backlash from the community". A 2022 paper on content moderation was retracted for similar reasons: the statement of removal [15] cites GDPR violations in the analyzed and redistributed user content. The focus on consent and protection of user content aligns with the developer’s earlier stance on search [16]. However, Mastodon introduced post search on the federated timeline as a per-account opt-in feature in September 2023 [17]. This new means of consent allows us to study post search on Mastodon in typical Cranfield experiments, especially under TIRA’s [5] protection of the index. TIRA as an EaaS platform allows the evaluation of retrieval systems on privately held indexes, where only the evaluation results reach the public.
Regarding the search for microblogs, we can build on substantial prior work on task definition
and evaluation from the TREC 2011–2015 Microblog tracks [18, 19, 20, 21, 22]. The tracks introduced
essentially two paradigms of tasks: (
Regarding the search in federated systems, we can build on prior work regarding tasks and, to a degree, systems and evaluation as presented in a recent survey of federated web search [29]. The TREC 2013–2014 Federated Web Search tracks [30, 31] introduce three tasks. The first task is “Resource Selection”, where users should determine the best resource to query given the results of several prior queries. The TREC track considers diferent search engines that are heterogeneous in content type, search collections, and retrieval systems. Mastodon, on the other hand, is almost homogeneous: all resources implement the same protocol (i.e., ActivityPub), all have the same content type (posts) and similar retrieval systems (although there might be diferent software versions). The challenges for resource selection instead are eficiency and politeness (see Section 4). The second task is “Results Merging”, where the given SERPs from various sources should be merged, and the relevance of the resulting list is scored. Since all posts can be ordered by post time, this task will be reduced to a microblog search. The third task is “Vertical Selection”, where the best set of topics, genres, and media types should be determined for each query. Although verticality might become relevant for search on Mastodon at some point (e.g. for balancing text, images, and videos in the results or to extend the search across the Fediverse), we decided to ignore it for now. For now, we assume that the resource selection aspect combined with ad-hoc microblog search form a well-grounded basis for a first evaluation campaign.
We created our collection of 733 million posts by capturing the local and federated timeline of 1,015 nodes
concurrently for 61 days, between December 12, 2023 and February 21, 2024. Our corpus is built in
three steps: (
There are many ActivityPub nodes on the Fediverse and many of them are small, inactive, or do not concern Microblogs. To limit the acquisition and filtering load, we sampled a subset of all discoverable nodes in a principled way. First, we got the 22,178 discoverable ActivityPub nodes through a public and up-to-date list.5 Second, we download each node’s general statistics6 and their weekly activity7 across the three months prior to our crawling and subsequently discarded the 11,822 non-Mastodon nodes without these endpoints. Third, from the remaining 10,354 candidate nodes, we sampled 1,000 Mastodon 5https://nodes.fediverse.party (crawler source code available online: https://github.com/Minoru/minoru-fediverse-crawler) 6https://nodeinfo.diaspora.software/ 7https://docs.joinmastodon.org/methods/node/#activity 101 103 105
101 103 105
nodes based on the six activity measures shown in Figure 1: (
The goal of our sampling strategy is to include most (moderately) large and active nodes while also representing smaller and less active nodes. However, the histograms in Figure 1 show that the activity across nodes is roughly log-normally distributed but skewed towards low activity, so less active nodes would dominate a (uniform) random or a (by size and activity) stratified sample. Instead, we apply weighted sampling, where the weight of each node ∈ is discounted by the joint probability of observing its activity statistics (); ∈ . We define a random variable for each of the six activity measures: 1 := total users, 2 := monthly active users, 3 := total posts, 4 := average weekly posts, 5 := avg. weekly logins, and 6 := avg. weekly registrations, where each variable is ∼ Lognormal( , 2). We fit the mean and standard deviation 2 of each log-normal distribution on the measures of the candidate nodes. For computational simplicity, we assume the independence of 1, . . . , 6, although that is generally not the case (see Figure 2). We calculate the sampling probability as: () ≈
∑︁ ∈ with =
1 (1, . . . , 6) and (1, . . . , 6) = ∏︁ () ∈
In other words, nodes whose activity scores are very likely under the log-normal distributions are less likely to be sampled. In the candidate nodes, this afects nodes with low activity much stronger and so the final samples (see Figure 1, yellow bars) are less skewed.
Online Resampling During the corpus construction, 15 nodes became unavailable to the crawler.8 They may have gone ofline, which is not uncommon for smaller nodes, or they might have blocked our crawler. Since this efect is relevant for resource selection, we decided to replace these nodes on the fly with the node with the closest sampling weight. Consequentially, the timelines of some nodes are only available until a time , and others are only available starting from . ◁ If node has not been crawled before, 0 is not set.
◁ Crawling has “catched up” with the timeline. ◁ Stop crawling, e.g., to continue with streaming crawling.
◁ Continue crawling the next batch.
◁ Run batched crawling until we “catch up” with the timeline.
Input: Mastodon node , Elasticsearch index . function crawl_batched(node , index , should stop early) is
Determine last crawled post 0 of from index if 0 is unknown then
else
Fetch next timeline batch after 0 from node . if batch is empty then if should stop early then
return for post in batch do if noindex flag is not set then
crawl_batched(, , should stop early) function crawl_streaming(node , index ) is crawl_batched(, , true) for post in timeline stream of do if noindex flag is not set then
Save post to index . if unrecoverable error has occurred then crawl_batched(, , false) return crawl_streaming(, ) ◁ Fall back to batch crawling.
◁ Start streaming/crawling.
Algorithm 1: Pseudocode of a crawler process for a single node. Crawling either uses Mastodon’s streaming or batch APIs. One crawler process is run for each of the 1,000 sampled nodes (see Section 3.1).
We implemented a parallelized, polite, and privacy-respecting crawler that fetches all public posts from each node’s federated timeline and stores the posts in an Elasticsearch index. Mastodon nodes ofer a streaming API that pushes new posts to listening applications via a long-lived HTTP or WebSocket connection,9 and a REST endpoint for fetching batches of posts.10 Our crawler uses the streaming API by default (since it causes less load on the nodes) and, if it is unavailable, falls back to the REST API (see Algorithm 1). We also use the REST API to fill “gaps” caused by eventual crawler downtimes. We schedule one crawler process for each of the 1,000 sampled nodes as self-restarting jobs on our Kubernetes cluster (1,620 CPU cores, 25 TB RAM). To balance the indexing load and to be future-proof (see Section 4), we create one index per month of crawling (i.e., December, January, and February) and distribute each index across 20 shards.
Considering politeness, our crawler identifies itself via a custom User-Agent,11 respects server-imposed wait times (i.e., Retry-After headers), and limits subsequent requests (e.g., due to connection errors) politely with an exponential backof (minimum: 14 seconds). Considering privacy, we explicitly remove all crawled posts without the users’ permissions, i.e. we remove posts where the noindex12 flag is set. Stored Data per Post The collection contains ca. 88 GB (index size) of post data per day across 61 days, allocating a total of 6 TB of storage. Table 1 shows an overview of the fields stored for each post. By default, we store all fields returned by the API using the exact same field names to maximize compatibility with Mastodon and its documentation. We also adopt Mastodon’s handling of optional and conditional fields: we omit empty optional fields like cards and store empty values for conditional 8List of the re-sampled nodes: https://github.com/webis-de/mastodon-search/blob/main/data/nodes_resample.txt 9https://docs.joinmastodon.org/methods/streaming/#public 10Up to 40 posts per request; https://docs.joinmastodon.org/methods/timelines/#public 11User agent: Webis Mastodon crawler (https://webis.de/, webis@listserv.uni-weimar.de) 12https://docs.joinmastodon.org/entities/Account/#noindex Field Fields of the post ifelds like reply_to and spoiler. We add 4 fields: the handle (username@node), the crawl date, the node from whose timeline the post was crawled, and a universally unique ID (based on hashing the crawled node and post ID) that is used for Elasticsearch indexing and random access.
We make our document collection accessible to other researchers on TIREx (see Section 2 for a discussion) by extending the ir_datasets library. We publicly release a development dataset of about 1,000 posts to demonstrate the export format. We export the collection via newline-delimited JSON files 13
The metadata of the attachment (duration, size, aspect ratio, bitrate, etc.). 107 s105 tn103 eem101 lfE 101 o reb107 um105 N103 101 100
102 103 104
101 102
101 102 103 100
101
100
101 102 103
101 102 103
100 101 102 103 104
where each line is individually compressed with GZIP.14 The files that contain the (compressed) posts are partitioned by node in individual directories and sub-partitioned in files of up to 1 GB (e.g., ./mastodon.social/0001.jsonl.gz) to allow for eficient filtering of nodes and in-memory reads of individual files. An index file that maps post IDs to the corresponding file location is included to allow for random access to individual posts by their IDs.
With ir_datasets, researchers can eficiently access the posts from all crawled nodes via the mastodon dataset ID, or just the post from a single node (e.g., via the mastodon/sigmoid.social dataset ID). In both cases, our ir_datasets extension implements random access by post IDs.
After crawling posts for 61 days, we have accumulated a total of 733 million posts from 1,015 nodes (due to re-sampling, see Section 3.1), allocating 6 TB of storage on our Elasticsearch cluster. Here, we quantitatively analyze several noteworthy properties of the document collection.
Text Length We measure the text length in tokens15 of the eight text fields that can appear in the
posts (see Figure 3). Most fields have a length as would be expected for microblogging: Most hashtags
have just one or very few tokens, account notes 100s, alt-texts 10s, and account names and spoiler text
in the single digits. However, there are some irregularities. First, there are many hashtags with 10s of
tokens, which are mostly Japanese and Chinese, where long phrases are regularly used as tags (e.g. “If
15 people call your name, they love you.”). Second, while most contents are in the typical range of 10s
or low 100s of tokens, the are also many posts with 1,000s of tokens, up to about 10,000. Upon manual
examination of a random sample of 10 posts with at least 1,000 tokens, we find that these posts are
mostly spam or contain excessively many hashtags (that also count towards the content length).
Contributing Nodes The timelines of our collected nodes contain many posts from remote nodes
(72% of all posts, see Table 1). A substantial part of these posts originates from 16,655 distinct Fediverse
nodes, despite us having only crawled from 1,015 Mastodon nodes (6%). Table 3 shows the source nodes
with the most contributed posts and their characteristics. Notably, half of the shown nodes are not
Mastodon servers themselves. Instead, misskey.io, live-theater.net, and misskey-square.net are nodes
from the Misskey16 microblogging network that is popular in Japan. The other two non-Mastodon
nodes (sportsbots.xyz and rss-parrot.net) are hubs for automated accounts only. Apart from mstdn.jp,
our sample (see Section 3.1) contained all of the top-10 contributing Mastodon nodes.
14https://datatracker.ietf.org/doc/html/rfc1952
15As indexed in Elasticsearch, see https://elastic.co/guide/en/elasticsearch/reference/current/token-count.html
16https://misskey-hub.net/en/
Node
mastodon.social
misskey.io
mstdn.jp
mstdn.social
live-theater.net
sportsbots.xyz
rss-parrot.net
fedibird.com
mastodon.online
Æ
✓
p
p
✓
p
p
p
✓
✓
misskey-square.net p
ø
✓
p
✓
✓
p
p
p
✓
✓
p
Nodes contributing the most posts to our corpus, whether they were sampled for crawling (Æ; see Section 3.1),
whether it is a Mastodon (ø) or other ActivityPub server, the covered timespan (z; 1% to 99% percentile), number
of unique posts, avg. redundancy (¨; same post crawled from multiple timelines), and proportions: reply posts (¾),
boosting posts (Â), posts with media (ë), with hashtags (º), with linked content (
Furthermore, 13% of the posts in our collection are replies, and 15% are boosts. About a fifth of the posts contain media attachments or hashtags, respectively. A quarter of the posts link to external content, while only a few posts contain spoilers (2%) or sensitive content (4%). Bot accounts contributed a fifth of all posts in our document collection. Most users did not opt-out from being indexed, and 61% of the posts were authored by accounts that explicitly opted in to search and discovery services. Locked accounts (i.e., follow requests manually approved) are most popular on non-Mastodon nodes (an extreme case being misskey-square.net; 40%), while sensitive content warnings are more popular on Mastodon nodes. The largest contributor to our document collection, mastodon.social closely approximates the post characteristics observed for the whole collection; the most notable diferences being a higher share of posts with linked content (37% vs. 24%) and a lower share of posts from locked accounts (6% vs. 12%). Frequent Hashtags, Languages, Accounts, and Apps Finally, Table 4 shows the most frequently used hashtags, languages, and applications, as well as the most active accounts in the document collection. The top hashtags are the typical tags for news (e.g., #news, #press) and hobbies (e.g., #nowplaying, #photography) besides some irregularities: The #nsfw hashtag (commonly used to indicate sensitive content) is relatively popular; it also seems to be somewhat common to mark bot posts with a hashtag (i.e., #bot); and compared to general news hashtags, the Russian invasion of Ukraine (i.e., #ukraine) is a dominant news topic. The language distribution is relatively diverse, with only about a third of the posts tagged as English, closely followed by Japanese (22%). Generally, the Fediverse seems to be popular in Japan, as also five of the top-10 nodes are Japanese. The remaining top-10 languages are European languages and Chinese.17 The users in our document collection use a large variety of applications to create posts. While many posts were authored on the Web (7%) or various Android or iOS apps, a large proportion of the posts also report which bot was used, which might be a useful ranking feature for search. Regarding user activity, we find that some accounts post excessively and the most “active” users contributed tens of thousands of posts to our collection. For example, the account “my24group” posts more than once every minute, on average. The top-7 most-posting accounts and “realTuckFrumper” are news bots; the remaining two are bots posting memes (“usluck”) or recipes (“dnc”).
Mastodon is highly similar to other microblogging platforms regarding content, interactions, metadata, and information needs but there are several unique constraints. We particularly look at how centrality and politeness influence task design and evaluation, how visibility and content policy influence the creation of collections, and how the federation influences the interaction features.
Each Mastodon node has two server-wide timelines: (
For a search task, this centrality efect has two implications. First, selecting the federated timeline of the largest or most central nodes is likely very eficient and efective. It may be Pareto optimal for small nodes to forward all search requests to the largest known node, which is not polite (see Section 4.2) and goes against the idea of a federated social network. Additionally, for users on very large nodes, it is likely eficient and efective to only search the node’s federated timeline, which would introduce a bias: it excludes small and isolated communities (which might have relevant expertise on a topic) and risks creating echo chambers. Hence, the diversity and specificity (i.e. selecting small but specialized nodes over large ones) should be considered in the task design and evaluation, either by penalizing the usage of federated timelines or by penalizing the reuse of large nodes for every topic.
Second, searching the federated instead of the local timeline will be more efective, since there are more documents in the index, but it will also be less efective since the index will be larger and there will be (many) duplicates across federated timelines which will have to be removed. This diference should be considered when evaluating eficiency.
The Fediverse is a large federated network with well over 10,000 nodes that could be queried. This means that both the network and per-node load would be extreme even if just 1% of the nodes are selected for each query. Minimizing the number of selected resources is especially important since most nodes may not be able to aford a steep increase in trafic or compute cost. This implies that a certain politeness is required and the evaluation must consider computational and network eficiency. This could be as simple as applying a penalty function on the number of selected resources instead of just scoring the ranking in a resource selection task. An alternative design could also consider the Pareto optima between efectiveness and eficiency. 17News reports suggest that Chinese users move to Mastodon to avoid censoring and punishment [32, 33].
However, rating the politeness via the number of selected resources more strongly promotes the problems of centrality (see Section 4.1), for example by only querying the one largest node. That means the eficiency measures need to penalize the (ab)use of few central nodes.
Two of the stand-alone features of Mastodon are post-visibility control and consent to process. Visibility can be controlled on 4 levels: public (visible to everyone), unlisted (visible to followers or via direct link), followers only, and direct messages. It is paramount to collect and index only public posts, which is the default for the public (not authenticated) API we used to create our collection.
Visibility is enforced through the public key authentication integrated into all Mastodon nodes. Since the origin nodes decide to which inbox any outgoing messages are going to be delivered, this authentication also controls visibility in case nodes de-federate or block certain actors. This is why a central search is unwelcome by many users: it would counteract these self-protection mechanisms by circumventing the node’s authentication.
Consent to process is expressed through two opt-in features that are included in every post’s ActivityPub message: discoverable and indexable (since Version 4.2). Discoverable indicates if the account can appear in discovery services like recommenders or user search. Indexable indicates if a post can be indexed and searched for. In our document collection, ca. 49% of posts did not opt-in to search and were not indexed and analyzed. An additional 35% of posts originated from non-Mastodon software which has no noindex flag and which we included in our index for quantitative analysis.
Mastodon timelines contain posts from diferent Mastodon nodes, but also from other Fediverse software (like Misskey, Lemmy, or Pixelfeed) that federate via ActivityPub (cf. Table 3). Since these are independently operated, the allowed content difers between nodes in, for example, maximum text length, desired topics, disallowed content or topics, or mandate for content warnings and alt-text. In addition, all moderation eforts are up to the node and bad actors may exist.
For a search task, the content policy has two implications. First, some nodes may more often produce relevant content because they allow longer posts, which makes those nodes more attractive for a retrieval system with detrimental efects on diversity (see Section 4.1) and politeness (see Section 4.2). Second, the search can not rely on the nodes for moderation. That means it can not ignore blocked or de-federated nodes or retrieve harmful content even though it exists in the sources. However, the harmful content can be removed from the document collections to separate moderation and search.
Interaction information, replies, boosts (reposts), and favorites (likes), are essential features for microblog search that behave diferently on Mastodon [ 34]. The most obvious is that favorites are not federated and each node keeps individual counters, except that every favorite is announced to the origin node. Similarly, boosts are only announced to the origin and the followers of the booster, so an node’s boost counter equates to the number of times a post has entered the node’s federated timeline. The propagation of replies is more reliable, although it is not guaranteed that all replies in a tree are the same for every node (refer to Jambor [34] for a precise explanation).
These diferences mean that posts from the origin node will have higher interaction counts than the same posts from the federated timeline, especially those of smaller nodes. Our collection does not contain any interaction data since we crawled new posts from the streaming API.
We have presented a new collection of microblog posts from the federated social media platform Mastodon to be used in information retrieval research. The collection contains about 733 million public posts from the federated timelines of 1,015 diverse Mastodon nodes across 61 days. We ofer access to the document collection in a privacy-preserving manner via TIREx and we provide our parallelized and polite Mastodon crawler as part of the code of this publication.
Analyzing the collection with respect to search on Mastodon, we identified several challenges for retrieval systems. First, the centrality of nodes will have a large efect on retrieval systems and their evaluation regarding efectiveness, eficiency, resource selection, and politeness. Second, visibility and consent is less of an issue than we previously assumed: 61% of the public, federated posts are opted into search, which already are over 10 million unique posts per month. Third, although microblogs on the Fediverse are structurally similar to those of well-researched sites, the content difers in length (even between nodes) and interaction statistics are unreliable.
There are two notable limitations to our collection. First, most post interaction statistics are missing since we collect the posts, usually, directly after they have been created. We might later add those statistics. Second, our document collection only consists of posts from the timelines of Mastodon nodes as, to our knowledge, no other Fediverse software implements a confirmation mechanism for search. If these mechanisms become available, we plan to extend our document collection with generic ActivityPub-compliant software and networks.18
Partially supported by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070014 (OpenWebSearch.eu). [8] L. La Cava, S. Greco, A. Tagarelli, Information consumption and boundary spanning in decentralized online social networks: The case of Mastodon users, Online Social Networks and Media 30 (2022). doi:10.1016/j.osnem.2022.100220. [9] A. Z. Rozenshtein, Moderating the Fediverse: Content moderation on distributed social media, J.
Free Speech L. 3 (2023) 217–235. URL: https://heinonline.org/HOL/P?h=hein.journals/jfspl3&i=217. [10] R. W. Gehl, D. Zulli, The digital covenant: Non-centralized platform governance on the Mastodon social network, Information, Communication & Society 26 (2023) 3275–3291. doi:10.1080/ 1369118X.2022.2147400. [11] L. La Cava, L. M. Aiello, A. Tagarelli, Get out of the nest! drivers of social influence in the #TwitterMigration to Mastodon, arXiv 2305.19056, 2023. doi:10.48550/arXiv.2305.19056. [12] U. Jeong, P. Sheth, A. Tahir, F. Alatawi, H. R. Bernard, H. Liu, Exploring platform migration patterns between Twitter and Mastodon: A user behavior study, arXiv 2305.09196, 2023. doi:10.48550/ arXiv.2305.09196. [13] J. He, H. B. Zia, I. Castro, A. Raman, N. Sastry, G. Tyson, Flocking to Mastodon: Tracking the great Twitter migration, in: Proceedings of IMC 2023, ACM, 2023, pp. 111–123. doi:10.1145/3618257. 3624819. [14] J. Trienes, A. T. Cano, D. Hiemstra, Recommending users: Whom to follow on federated social networks, arXiv 1811.09292, 2018. doi:10.48550/arXiv.1811.09292. [15] Statement of removal. Mastodon content warnings: Inappropriate contents in a microblogging platform, in: Proceedings of ICWSM 2022, volume 13, 2022. URL: https://ojs.aaai.org/index.php/ ICWSM/article/view/22003. [16] E. Rochko, Cage the Mastodon: An overview of features for dealing with abuse and harassment, 2018. URL: https://blog.joinmastodon.org/2018/07/cage-the-mastodon/. [17] E. Rochko, Mastodon 4.2: A new search experience and more!, 2023. URL: https://blog.joinmastodon.
org/2023/09/mastodon-4.2. [18] I. Ounis, C. Macdonald, J. Lin, I. Soborof, Overview of the TREC 2011 microblog track, in: Proceedings of TREC 2011, volume 500-296 of NIST Special Publication, NIST, 2011. URL: https: //trec.nist.gov/pubs/trec20/papers/MICROBLOG.OVERVIEW.pdf. [19] I. Soborof, I. Ounis, C. Macdonald, J. Lin, Overview of the TREC-2012 microblog track, in: Proceedings of TREC 2012, volume 500-298 of NIST Special Publication, NIST, 2012, p. 7. URL: https://trec.nist.gov/pubs/trec21/papers/MICROBLOG12OVERVIEW.pdf. [20] J. Lin, M. Efron, Overview of the TREC-2013 microblog track, in: Proceedings of TREC 2013, volume 500-302 of NIST Special Publication, NIST, 2013. URL: https://trec.nist.gov/pubs/trec22/ papers/MB.OVERVIEW.pdf. [21] J. Lin, Y. Wang, M. Efron, G. Sherman, Overview of the TREC-2014 microblog track, in: Proceedings of TREC 2014, volume 500-308 of NIST Special Publication, NIST, 2014. URL: https://trec.nist.gov/ pubs/trec23/papers/overview-microblog.pdf. [22] J. Lin, M. Efron, G. Sherman, Y. Wang, E. M. Voorhees, Overview of the TREC-2015 microblog track, in: Proceedings of TREC 2015, volume 500-319 of NIST Special Publication, NIST, 2015. URL: https://trec.nist.gov/pubs/trec24/papers/Overview-MB.pdf. [23] J. Lin, A. Roegiest, L. Tan, R. McCreadie, E. M. Voorhees, F. Diaz, Overview of the TREC 2016 real-time summarization track, in: Proceedings of TREC 2016, volume 500-321 of NIST Special Publication, NIST, 2016. URL: https://trec.nist.gov/pubs/trec25/papers/Overview-RT.pdf. [24] J. Lin, S. Mohammed, R. Sequiera, L. Tan, N. Ghelani, M. Abualsaud, R. McCreadie, D. Milajevs, E. M. Voorhees, Overview of the TREC 2017 real-time summarization track, in: Proceedings of TREC 2017, volume 500-324 of NIST Special Publication, NIST, 2017. URL: https://trec.nist.gov/ pubs/trec26/papers/Overview-RT.pdf. [25] R. Sequiera, L. Tan, J. Lin, Overview of the TREC 2018 real-time summarization track, in: Proceedings of TREC 2018, volume 500-331 of NIST Special Publication, NIST, 2018. URL: https: //trec.nist.gov/pubs/trec27/papers/Overview-RTS.pdf. [26] J. Rao, W. Yang, Y. Zhang, F. Türe, J. Lin, Multi-perspective relevance matching with hierarchical
ConvNets for social media search (2019) 232–240. doi:10.1609/AAAI.V33I01.3301232.