This research relates to past work in using AI to increase participants’ motivation in citizen science research as well as work in aplying recommendation systems in real world settings. We list relevant work in each of these two areas. 1.1

Online participation in citizen science projects has become very common [ 21 ]. Yet, most of the contributions rely on a very small proportion of participants [ 25 ]. In SciStarter, the group of participants who contribute to more than 10 projects is less than 10% of all users. However, in most citizen science projects, the majority of participants carry out only a few tasks. Many researches have explored the incentives and motivations of participants in order to increase participants engagement. Kragh et al. [ 15 ] showed that participants in citizen science projects are motivated by personal interest and desire to learn something new, as well as by the desire to volunteer and contribute to science. A prior work of Raddic et al. [ 23 ] also showed that participants engagement has mainly originated in pure interest in the project topic, such as astronomy and zoology. Yet, as we tested this finding in our collected data, we noticed that user interest is very diverse and does not include only one major topic of interest. Nov et al. [ 21 ] explored the diferent motivations of users to contribute, by separating this question to quantity of contribution and quality of contributions. They showed that quantity of contribution is mostly determined by the user interest in the project and by social norms while quality of contribution is determined by understanding the importance of the task and by the user’s reputation. In our work we aimed to increase only the quantity of contributions, since data about the quality of contribution is not available for us.

A significant prior work was done in order to increase participants engagement, which takes into consideration user motivation as well. Segal et al. [ 29 ] have developed an intelligent approach which combines model-based reinforcement learning with of-line policy evaluation in order to generate intervention policies which significantly increase users’ contributions. Laut et al. [ 17 ] have demonstrated how participants are afected by virtual peers and showed that participants’ contribution can be enhanced through the presence of virtual peers.

Ponciano et al. [ 22 ] characterized volunteers’ task execution patterns across projects and showed that volunteers tend to explore multiple projects in citizen science platforms, but they perform tasks regularly in just a few of them. They have also showed that volunteers recruited from other projects on the platform tend to get more engaged than those recruited outside the platform. This ifnding is a great incentive to increase user engagement in SciStarter’s platform instead of in the projects’ sites directly, like we do in our research.

In this research, we attempted to enhance participant engagement with citizen science projects by recommending the user projects which best suit the user preferences and capabilities. 1.2

Similar to our work, other researchers, also tried to increase user engagement and participation by personalized recommendations. Labarthe et al. [ 16 ] built a recommender system for students in MOOCs that recommends relevant and rich-potential contacts with other students, based on user profile and activities. They showed that by recommending this list of contacts, students were much more likely to persist and engage in MOOCs. A subsequent work of Dwivedi et al. [ 7 ] developed a recommender system that recommends online courses to students based on their grades in other subjects. This recommender was based on collaborative filtering techniques and particularly item based recommendations. This paper showed that users who interacted with the recommendation system increased their chance to finish the MOOC by 270%, compared to users who did not interact with the recommendation system.

Some other studies that concern user engagement with recommendations systems showed how early intervention significantly increase user engagement. Freyne et al. [ 9 ] showed that users who received early recommendations in social networks are more likely to continue returning to the site. They showed a clear diference in retention rate between the control group, which has lost 42% of the users, and a group that interacted with the recommendations, which has lost only 24% of the users.

Wu et al. [ 32 ], showed how tracking user’s clicks and return behaviour in news portals succeeds to increase user engagement with their recommendation system. They formulated the optimization of long-term user engagement as a sequential decision making problem, where a recommendation is based on both the estimated immediate user click and the expected clicks results from the users’ future return.

Lin et al. [ 18 ], developed a recommendation system for crowdsourcing which incorporates negative implicit feedback into a predictive matrix factorization model. They showed that their models, which consider negative feedback, produce better recommendations than the original MF approach of implicit feedback. They evaluated their findings via experiment with data from Microsoft’s internal Universal Human Relevance System and showed that the quality of task recommendations is improved with their models. In our work, we use only positive implicit feedback, due to the low users trafic, where a significant evidence of negative feedback is hard to be found.

Recommendation algorithms are mostly evaluated by their accuracy. The underlying assumption is that accuracy will increase user satisfaction and ultimately lead to higher engagement and retention rate. However, past research has suggested that accuracy does not necessarily lead to satisfaction. Wu et al [ 31 ] investigated the efects of popular approaches such as collaborative-filtering and content-based to see if they have diferent efects on user satisfaction. Results of the study suggested that product awareness (the set of products that the user is initially aware of before using any recommender system) plays an important role in moderating the impact of recommenders. Particularly, if a consumer had a relatively niche awareness set, chances are that content based systems would garner more positive responses on the satisfaction of the user. On the other hand, they showed that users who are more aware of popular items, should be targeted with collaborative filtering systems instead. A subsequent work of Nguyen et al [ 20 ], showed that individual users’ preferences for the level of diversity, popularity, and serendipity in recommendation lists cannot be inferred from their ratings alone. The paper suggested that user satisfaction can be improved by integrating users’ personality traits into the process of generating recommendations, which were obtained by a user study.

Our goals for the research project were to (1) help users discover new projects in the SciStarter ecosystem - matching them with projects that are suitable to their preferences. (2) learn user behavior in SciStarter, and develop a recommendation system which will help increase the number of project they contribute to. (3) measure users’ satisfaction with the recommendation system.

We adopted several canonical algorithms from the recommendation systems literature: CF user based [ 28 ], CF item based [ 28 ], Matrix Factorization [ 27 ], Popularity [ 1 ]. These approaches were chosen as they are all based on analyzing the interactions between users and items and do not rely on domain knowledge which is lacking (such as project’s location, needed materials, ideal age group etc.). Each algorithm receives as input a target user and the number of recommendations to generate (N). The algorithm returns a ranking of top N projects in decreasing order of relevance for the user. We provide additional details about each algorithm below. 2.0.1 User-based Collaborative Filtering. In this algorithm, the recommendation is based on user similarities [ 28 ]. The ranking of a project for a target user is computed by comparing users who interacted with similar projects. We use a KNN algorithm [ 6 ] to ifnd similar users, where the similarity score for user vector U1 and user vector U2 from the input matrix, is calculated with cosine similarity.

1 ∗ 2 ( 1, 2) = || 1|||| 2|| We chose the value of to be the minimal number such that the number of new projects in the neighborhood of similar users to the target user equaled the number of recommendations. In practice was initially chosen to be 100 and increased until this threshold was met. This was done so that there will always be suficient number of projects to recommend for users. 2.0.2 Item-based Collaborative Filtering. In this algorithm the recommendation is based on project similarity [ 28 ]. The algorithm generates recommendations based on the similarity between projects calculated using people’s interaction with these projects. Similarity score for project vector P1 and project vector P2 from the input matrix, is calculated with cosine similarity.

1 ∗ 2 ( 1, 2) = ( ) = || 1|||| 2||

The algorithm then recommends on the top-N most similar projects to the set of projects the user has interacted with in the past. 2.0.3 Matrix Factorization - SVD. The Matrix factorization algorithm (SVD) directly predicts the relevance of a new project to a target user by modeling the user-project relationship [ 14, 27 ]. This model-based algorithm (as opposed to the two memory based algorithms presented earlier) was chosen since it is one of the leading recommendation system algorithms [ 11, 14, 26 ]. SVD uses a matrix where the users are rows, projects are columns, and the entries are values that represent the relevance of the projects to the users. This users-projects matrix is often very sparse and has many missing values, since users engage with a very small portion of all the available items.

The algorithm estimates the relevance of a target project for a user by maintaining a user model and a project model that include hidden variables (latent factors) that can afect how users choose items. These variables have no semantics, they are simply numbers in a matrix; in reality, aspects like gender, culture, age etc. may afect the relevance, but we do not have access to them.

The singular value decomposition (SVD) of any matrix is a factorization of the form . This algorithm is used in recommendation systems in order to find the multiplication of the three matrices , , , to estimate the original matrix and hence, to predict the missing values in the matrix. As mentioned above, the matrix includes missing values as users did not participate in all projects. We estimate the missing values which reflect how satisfied will the user be with an unseen project. In the settings of recommendation system, the matrix is a left singular matrix, representing the relationship between users and latent factors. is a rectangular diagonal matrix with non-negative real numbers on the diagonal, while is a right singular matrix, indicating the similarity between items and latent factors. SVD decreases the dimension of the utility matrix , by extracting its latent factors. It maps each user and item into a latent space with r dimensions and with this, we can better understand the relationship between users and projects, and compare between their vectors’ representations. Let Rˆ be the estimation of the original matrix R. Given Rˆ, which includes predictions for all the missing values in R, we can rank each project for a user, by its score in Rˆ. The projects with the highest ranking are then recommended to the user. In our settings, like in the other algorithms described before, Rˆ is a binary matrix. 3

The first part of the study compares the performance of the diferent algorithms on historical SciStarter Data. The second part of the study implements the algorithms in the wild, and actively assigns recommendations to users using the diferent algorithms.

Of the 3000 existing projects SciStarter ofers, 153 projects are afiliate projects. An afiliate project is one that uses a specific API to report back to SciStarter each time a logged in SciStarter user has contributed data or analyzed data on that project’s website or app. As data of contributions and participation only existed for the afiliate projects, we only used these projects in the study. 3.1

The training set for all algorithms consisted of data collected between January 2012 to September 2019. It included 6353 users who contributed to 127 diferent projects. For the collaborative filtering and SVD algorithm, we restricted the training set to users that made at least two activities during that time frame, whether contributing to a project or interacting on the SciStarter portal. We chronologically split the data into train and test sets such that 10% of the latest interactions from each user are selected for the test set and the remaining 90% of the interactions are used for the train set. As a baseline, we also considered an algorithm that recommends project according to decreasing order of popularity, measured by the number of users who contribute to the project [ 1 ].

We evaluate the top-n recommendation result using precision and recall metrics with varying number of recommendations.

Fig 4 shows results of the precision and recall curves for the 4 examined algorithms. As can be seen from the figure, user-based collaborative filtering and SVD are the best algorithms and their performance is higher than Popularity and Item based collaborative ifltering. The Popularity recommendation algorithm generated the lowest performance. The second part of the study was an online experiment. Users who logged on to SciStarter starting on December 2nd, 2019 were randomly assigned to one of 5 cohorts, each providing recommendations based on diferent algorithm: (1) Item-based Collaborative Filtering, (2) User-based Collaborative Filtering, (3) Matrix Factorization, (4) Most popular projects, (5) Promoted projects. Promoted projects were manually determined by SciStarter and often aligned with social initiatives and current events. Among these projects are GLOBE Observer Clouds4, Stream Selfie 5 and TreeSnap 6. Another example is FluNearYou, in which individuals report flu symptoms online, was one of the promoted projects during the COVID-19 outbreak. These projects change periodically by the SciStarter administrators.

The recommendation tool was active on SciStarter for 3 months. Users who logged on during that time were randomly divided into cohorts, each receiving a recommendation from a diferent algorithm. Each cohort had 42 or 43 users. The recommendations were embedded in the user’s dashboard in decreasing order of relevance, in sets of three, from left to write. Users could scroll to reveal more projects in decreasing or increasing order of relevance. Figure 5 shows the top three recommended projects for a target user.

All registered users in SciStarter received notification via email about the study, stating that the “new SciStarter AI feature provides personalized recommend projects based on your activity and interests.” A link to a blog post containing more detailed explanations of recommendation algorithms, their role in the study, emphasizing that‘ ‘all data collected and analyzed during this experiment on SciStarter will be anonymized." Also, users are allowed to opt 4https://scistarter.org/globe-observer-clouds 5https://scistarter.org/stream-selfie 6https://scistarter.org/treesnap out of receiving recommendations at any point, by clicking on the link “opt out from these recommendations" in the recommendation panel. In practice, none of the participants selected the opt out option at any point in time.

Figure 6 (top) shows the average click trough rate (defined as the ratio recommended projects that the users accessed) and Figure 6 (bottom) shows the average hit rate (defined as the percentage of instances in which users accessed at least one project that was recommended to them). As shown by the Figure, both measures show a consistent trend, in which the user-based collaborative algorithms achieved the best performance, while the baseline method achieving worse performance. Despite the trend, the diferences between conditions were not statistically significant in the < 0.05 range. We attribute this to the fact that we measured clicks on recommended projects rather than actual contributions which is the most important aspect for citizen science.

To address this gap we defined two new measures that consider the contributions made by participants to projects, which considers the system utility and identified by Gunawardana and Shani [ 12 ]. The measures include the average number of activities that users carried out in recommended projects (RecE), and the average number of activities that users carried out in non-recommended projects (NoRecE). Figure 7 compares the diferent algorithms according to these two measures. The results show that users assigned to the intelligent recommendation conditions performed significantly more activities in recommended projects than those assigned to the Popularity and Baseline conditions. Also, users in the SVD algorithm performed significantly less activities in non-recommended projects than the Popularity and Baseline conditions. These results were statistically significant according to Mann-Whitney tests (see Appendix for details).

Lastly, we measure the average number of sessions for users in the diferent conditions, where sessions are defined as a continuous length of time in which the user is active in a project. Figure 8 shows the average number of sessions for users in the diferent cohorts, including the number of sessions for the historical data used to train the algorithms, in which no recommendations were provided. The results show that users receiving recommendations from the personalized algorithms performed more sessions than the number of sessions in historical data. These results are statistically significant. Although there is a clear trend that users in the condition achieved the highest number of sessions, these results were not significant in the < 0.05 range.

To explain the success of SVD’s good performance in the online study, we note first that SVD is considered as a leading algorithm in the domain of recommendation systems [ 11, 26 ]. Second, in our setting SVD tended to generate recommendations that participants had not heard about before, which the survey reveals to be more interesting to them. One participant remarked: "I did not click on either project because I have looked at both projects (several times) previously", "I am more interested in projects I didn’t know exists before".

Lastly, we note the obstacles we encountered when carrying out the study. The first obstacle we encountered was the small number of relevant projects that could be recommended. Out of almost 3000 projects that SciStarter ofers, we restricted ourselves to about 120 projects are afiliate projects which actively provide data of users’ interactions. Another obstacle was that we were constrained to a subset of users who log on to SciStarter and use it as a portal to contributing to the project, rather than accessing the project directly. Out of the 65,000 registered users of SciStarter, only a small percentage are logged in to both SciStarter and an afiliate project. As a result, we have relatively few users getting recommendations. In addition, some of SciStarter’s projects are location-specific and can only be done by users in the same physical location. (e.g collecting a water sample from a particular lake located in a particular city). Therefore, we kept track of users’ location and restricted our recommendation system to be a location-based system, which recommends users with projects they are able to participate in. 3.3

In order to learn what is the users’ opinion on the recommendations, and their level of satisfaction, we conducted a survey with SciStarter’s users. Our survey was sent to all SciStarter community users. 138 users have filled the survey, where each user was asked about the recommendations presented to him by the algorithm he was assigned to. The survey included questions about users’ overall satisfaction with the recommendation tool as well as questions about their pattern of behavior before and after the recommendations. The majority of users (75%) were very satisfied with the recommendation tool and claimed that the recommendations matched their personal interests and goals. The majority of users (54%) reported they have clicked on the recommendations and visited the project’s site, while only 8% of users did not click the recommendation or visited the project site. Interestingly, users who were not familiar with the recommended projects before, clicked more on the recommendations, as well as users who previously performed a contribution to a project.

Users who did not click on the recommendations can be divided into 3 main themes: (1) Users who don’t have the time right now or will click the project in the future. (2) Users who feel that the recommendations are not suitable for their skills and materials: "Seemed out of my league", "I didn’t have the materials to participate". This behaviour was also discussed in [ 30 ], and was named "classification anxiety". (3) Users who feel that the recommendations are not suitable for their interests: "No interest in stall catchers", "The photos and title didn’t perfectly match what I am looking for".

The survey provided evidence for the positive impact of using the recommendation systems in SciStarter, which include the following comments. “I am very impressed by the new Artificial Intelligence feature from SciStarter! Your AI feature shows me example projects that I didn’t know before exist", and “I like how personalized recommendations are made for citizen science users". 4

This work reports on the use of recommendation algorithms to increase engagement of volunteers in citizen science, in which volunteers collaborate with researchers to perform scientific tasks. These recommendation algorithms were deployed in SciStarter, a portal with thousands of citizen science projects, and were evaluated in an online study involving hundreds of users who were informed about participating in a study involving AI based recommendation of new projects. We trained diferent recommendation algorithms using a combination of data including users’ behavior in SciStarter as well as their contributions to the specific project. Our results show that using the new recommendation system led people to contribute to new projects that they had never tried before and led to increased participation in SciStarter projects when compared to a baseline cohort group that did not receive recommendations. The outcome of this research project is the AI-powered Recommendation Widget which has been fully deployed in SciStarter. This project has transformed how SciStarter helps projects recruit and support participants and better respond to their needs. It was so successful in increasing engagement, that SciStarter has decided to make the widget a permanent feature of their site. This will help support deeper, sustained engagement to increase the collective intelligence capacity of projects and generate improved scientific, learning, and other outcomes. The results of this research have been featured on the DiscoverMagazine.com 7. While we observed significant engagement with the recommendation tool, one may consider adding explanations to the recommendations in order to increase the system’s reliability and user’s satisfaction with it. Moreover, we plan to extend the recommendation system to include content based algorithms, and test its performance as compared to the existing algorithms. We believe that integrating content in Citizen Science domain can be very beneficial. Even though users tend to participate in a variety of diferent projects, we want to be able to capture more intrinsic characteristic of the projects, such as the type of the task a user has to perform or the required efort. 7https://www.discovermagazine.com/technology/ai-powered-smart-projectrecommendations-on-scistarter

A A.1

A Mann-Whitney test was conducted to compare between each condition in the online experiment. Table 1 presents the results of the pairwise tests for the measures RecE and NoRecE that are significant.