Although some online communities have been able to produce high quality products and to engage thousands of users, community designers usually struggle to engage new users and increase the level of contribution of current users. Some researchers have explored approaches to persuade users to collaborate. An important strand of this research area is based on sending messages to the current users and manipulate the content of the message in order to evaluate their e ectiveness. Mentioning the bene ts of contributing has been tested, however the results of di erent studies have been contradictory. One of them have reported a positive e ect in the contribution rate, but other one found that mentioning bene ts has depressed the level of contribution. Our hypothesis is that the e ectiveness of messages may depend on other users' variables and not in the content message only. To test our hypothesis, we performed a study to evaluate the e ect of messages mentioning community and personal bene ts in di erent users' cohorts. Levels of previous participation in the system and demographic data were tested in order to explain di erences in the e ectiveness of this engagement strategy.
Several well-known online communities have demonstrated the potential of producing high quality products, enable people all around the world to share content or collaborate in geographically distributed teams. However, many other online community projects have failed in engaging enough users to achieve critical mass. Researchers have explored di erent ways to nd out what motivates users to contribute, and how to increase their levels of contribution. Previous work has been mainly focused on using messages and manipulate the message content in order to encourage people to contribute to the community. Some studies have reported the e ect of mentioning the bene ts of contributing as a motivator, however the results in di erent studies has been contradictory. Mentioning the value of contributions has increased the level of contributions in one study, but it has decreased the contribution rate in another one.
We think that these contradictory results hint that the impact of a message may be a ected by users' characteristics, not just by the message content itself. Users may have di erent motivations to collaborate, so di erent strategies that match with these diverse motivations may generate more e ective results. These observations motivated us to explore adaptive engagement mechanisms in online communities. Our overall goal is to explore several ways of adaptation such as adapting to demographic data of users, user knolwedge, past levels of contribution, and the navigation patterns.
This paper reports our attempt to evaluate the e ectiveness of adaptation to
one aspect of user demography: user cultural background. Our initial hypothesis
is that the e ect of appealing to private vs. community bene ts may be di erent
for users with di erent cultural backgrounds. For example, given the popular
belief that people from Asian countries are more community-oriented, they might
get more motivated to work for community goals. In contrast, people from
Western countries are more concerned with personal bene ts and thus could be better
motivated to do work for their own goals. This popular belief has been also
supported by a multinational survey in [
The rest of the paper is organized as follows: Section2 will describe general background about online communities, and related work on using bene ts in the content of engagement messages. Section 3 will present the study design and the system that was used as testbed; Section 4 will detail the results of the study; Section 5 will include the discussion and future work and Section 6 will present the conclusions. 2 2.1
The term online community was rst de ned by Rheingold in 1994 [
Along with these well-known online communities, many others starting online
communities were never able to take o . Only 10.3 % of the Open Source projects
that have been created in SourceForge have more than three members [
Several research groups have focused their e orts on nding out ways to
maintain online communities alive longer. Several strands of work have been
studied such as:
{ how to socialize newcomers [
One of the main strands of research has focused on how to encourage
contributions. The main goal is to create the required amount of content (e.g. videos in
Youtube, pages in Wikipedia, code in Open Source systems) to provide bene ts
to the whole online community, including casual visitors. Section 2.2 will details
several ndings related to encouraging contributions to online communities.
Simply asking by contributions is the most popular strategy. Several di erent ways
to do so has been reported:
{ broadcasting an email asking for contribution [
In 2004, Beenen et al. [
Another study in MovieLens [
We believe that the reason to explain this contradictory results might be related to user's characteristics and its sensitivity to the kind of bene ts that were mentioned in the messages, more that to the content itself. 3
Building upon current knowledge in the e ectiveness of messages to encourage contributions, this study tested the e ect of sending emails with di erent information to users with di erent cultural background and di erent levels of participation. 3.1
We used CourseAgent system and its users as testbed of our studies. CourseAgent
[
Membership is restricted to the iSchool graduate students only. A new account is created for each new student who is enrolled in a graduate program at the iSchool. Recently, the system started to record when the students get their degree. So, there is partial knowledge about the student status. When we started the studies there were 1256 registered users. 123 users were already graduated according to the data in the system, 517 user had unknown student status and 616 were current students.
Out of 1256 registered users, 175 users (13,9%) have added at least one taken course to their study history. This is the most popular kind of contribution, others were done by fewer users. By the volume of contributions, the most successful feature is adding course evaluationsin in respect to a speci c career goal. There were 1085 contributions of this kind. These numbers show that CourseAgent is a community that is in its early stages, and that is has not achieved a high number of contributions yet. 3.2
The study was designed to test the impact of messages appealing to community bene t versus messages appealing to a personal bene t to the behavior of students with di erent cultural background. The sample was a subset of current iSchool graduate students. The cultural background of students was modelled by their home country (represented as a part of student demographic data). The impact was measured by monitoring the changes in the database (such added course ratings) and tracking user actions through the system log mechanism. The latter allowed to observe the the level of previous and current users activity in the system even for the users who havent contributed any information that is stored the system database.
The experiment manipulated the kind of message and the cohorts that received each message. A user only received one message during the study, and the users activities before and after getting the message were tracked and analyzed. Cohorts were de ned according an equally distributed users home country and the level of participation in the system before the message was sent.
The rst execution of the study was run during Fall 2010, when the Spring term registration period begun. The message asked users to rate 3 courses in they have taken before Fall 2010, thus all the users who have started their programs in Fall 2010 were removed from the subjects sample. The second round of emails was sent after the end of the Fall 2010 semester (but before Spring 2011 registration is nished) to users who had started their programs in Fall 2010, so they were now able to rate courses they took in their rst term. The messages that were sent in these two rounds are shown in Table 1.
The study was replicated in a slightly di erent form with newcomers. Students whose start term was Spring 2011 received a welcome email that mentioned community or personal bene t of contribution and asked to provide career goals and courses to be taken.
In total, e-mail messages were sent to 574 users. Six students received duplicated emails because they were students in the iSchool before, but started a new program in Fall 2010 or Spring 2011 so they were considered twice in the subject selection of di erent executions of the study. These users were removed from the analysis. Community Bene t Message CourseAgent enables the students to receive recommendations from other students, as well as advice from faculty, regarding their course of study, workload, and relevance of courses. The usefulness of CourseAgent recommendations for the student community increases as users provide more information including courses they have taken, their career goals, and their ratings of courses.
We are trying to enhance the utility of CourseAgent before Spring registration starts. Please help your fellow students by adding and rating three courses you have taken and completed in the past by November 22th. Your contribution will empower the system to better recommend courses to all of the iSchool students just in time for their Spring registration.
Private Bene t Message CourseAgent helps you to plan your course of study wiser by keeping track of your progress towards selected career goals and by o ering advice from faculty and peer students about workload and relevance of courses. The usefulness of CourseAgent increases as you provide more information about courses taken, career goals, and your ratings of courses.
We are trying to provide the best support for you before you start your Spring registration. To help us with that, please add and rate three courses you have taken and completed in the past by November 22th. Providing three course ratings by November 22th will help the system to present you a more complete picture of your progress (through the Career Scope tab) and better recommend you relevant courses just in time for your Spring registration.
The students who received these messages came from 30 di erent home countries to pursue their graduate degrees in the iSchool. Note that in our context, the home country is not just a country of birth, but a country where students lived and studied at least until nishing their high school. Moreover, with just a few exceptions, home country is also the country where iSchool graduate students received their undergraduate degree. As a result, in this context, student home country was used as reasonable indication of students cultural background. For this study, 6 groups of countries were de ned considering their geographic and cultural similarities, and the number of iSchool students who came from those countries. The categories were de ned as follows: { Unde ned: Students whose home country was not available at the moment of the study. { United States: Students whose home country is United States. { Asia: Students whose home country is China (PRC), Taiwan, Republic of
Korea, Japan, or Thailand. { India: Students whose home country is India. { Middle East: Students whose home country is Islamic Republic of Iran, Turkey, Saudi Arabia, Kuwait, or Egypt.
As a result of the study, 32 out of 568 message receivers used the system within one week after receiving the encouragement message (0.056%): 18 students who received the community bene t message and 14 who received the personal bene t message. Table 2 shows a detailed description of the results by country category. In our analysis of engagement we distinguished contributions (i.e. adding taken or planned courses and evaluation registration of courses) and actions that included both contributory actions and exploratory actions such as navigation through pages. Contributions add new information to the "community wisdom" and can measure the community-bene tial part of user engagement while the total volume of actions measures overall user engagement into working with the system. As the table shows, overall, the community message generated more actions in the system and more contributions.
Unknown Asia India Middle East Other US Total
The goal of the study was to test if the community bene t message could be more e ective in people from Asian countries, and the personal bene t message more e ective when sent to students from Western countries. Table 3 compares the numbers related to these two speci c cohorts. To our surprize, bottom-level data showed the opposite e ect - community bene t message engaged more users and produced more contributions among US students while personal bene t message engaged more Asian students and produced more contributions. However, a detailed analysis of the level of actions does not produce a clear picture. Asian users who received the community message executed more actions and contributed more to the system than Asian students who received the private message. US users provided a similar level of contribution and action when receiving the community bene t and the personal bene t message. A factorial logistic regression was run considering country category and kind of message as factors, and the fact of visiting the site within a week as the dependent variable. Although it seems that community message was able to engage more US and the personal bene t message engaged more Asian users, the predictor model using these factors didnt t signi cantly better than the null model. However, the study results were still able to show signi cant di erences in more speci c cases that will be described below.
Since the number of contributions and actions do not behave normally according to the normality tests, non-parametric tests were used to assess the signi cance of the di erence of mean number of actions in di erent cohorts. All of the following reported results are based on non-parametric tests.
Table 4 illustrates the gures related to engaged users only. Asian students executed more actions in the system than US students for both kind of messages (p<.049), however the di erence regarding number of contribution was not signi cant. Furthermore, users who have already contributed to the system provide signi cantly more contributions than newcomers (i.e. this includes current students who hasn't used the system before as well as new students - "No, but new" in the Table) (p<.003).
Table 5 shows the mean number of actions executed for users with di erent characteristics and the signi cance of mean di erences considering the whole sample, not only engaged users. The number of actions executed for users who received the community bene t message was higher the number of actions done by those who received private value. The number of contributions was also higher, however these di erences were not signi cant. Since 53 out of 568 emails were sent to new students, and 9 of them were nally engaged in using the system (7 community message and 2 personal message). The mean actions of this sample is much higher than the other 2 cohorts: current student who haven't visited the system and those who have visited the system before (p<.001). This can be explained by the information needs of new students. New students usually require to get as much information as possible to make decisions, however most of them are recently arriving to the city so they do not have enough social contacts to get all the required information. The system o ers them easy to access information about courses, and they spent most of the time looking for data in the system. However, they contribute less than current students. They do not have enough knowledge about courses to share, so their navigation pattern is more focused on browsing than contributing. Users who have contributed before to the system also contributed more after the message (p<.001). This can be related to the perception that the new time investment for contributing is low due to they have already invested time time in the system before. They just need to partially update their pro les in order to get the bene ts. On the other side, newcomers have to invest more time in the system to achieve the same bene ts.
Regarding the students who received the community bene t message, only the previous fact of lurking or contributing to the system were factors with a statistically signi cant di erences in the level of activity. However, as the results have suggested before, the mean number of actions and contributions are higher than those computed when considering both messages. Table 6 shows these gures.
The analogous analysis for students who received the personal bene t message was executed, and the the fact of contributing to the system before is the only factor that is signi cant in this case. See Table 7 for a detailed description of the data. 5
Our hypothesis that community bene t message will be more e ective with Asian students and the personal bene t will engage more US students was not con rmed. Unexpectedly, we found that the message appealing to the community Variable
Values Kind of Community Message Personal User has vis- No ited the system before Yes
No, but new User has con- No tributed information Yes before Home Country
Asia India Middle East US Female
Male Gender bene t message engaged more US students than the personal one, however the personal message engaged more Asian students. Although these di erences were not signi cant, the pattern is surprising and we plan to continue replicating the study to verify it. We did found, however, one signi cant di erence related to the demography: Asian users executed signi cantly more actions and added more contributions in the system than the US students without regard of the kind of message they receive.
At the same time, we found a few important di erences related not to user demograpy, but to their past experience and status in the system. Most importantly, users who have contributed to the system before contributed signi cantly more that the newcomers in the system. We think this is due to the fact that these users need to invest less time to improve their user pro les and get the bene ts of the system. On the other side, newcomers can be discouraged by the fact that they to create their pro le before getting personalized recommendations, so they quickly decide to stop contributing and start looking for useful information that can be obtained without a complete user pro le.
Being a new student is also a signi cant factor of the number of actions to be executed in the system. Regarding the entire samples (not only engaged users), new students execute signi cantly more actions than the other cohorts. However, they do not contribute more than the others. We believe that this re ects an information seeking behavior. As new students they probably lack if information as well as social contacts within the iSchool, so the system o ers them a way to explore information that they might need. However, they do not have enough information to share yet. We see this as an opportunity. We think that engaging new students might be easier that re-engaging those that have already decide not to use the system. 6
Online community designers usually struggle to encourage users to contribute enough content to make the site sustainable. One of the most common engagement mechanisms is to send messages to current users asking for contributions. Previous research has used the salience of bene ts in the message as a motivator, however this has produced contradictory results in di erent studies. In this paper, we proposed that the di erence could be explained by users' characteristics more than the message itself. We designed an experimental study to test the effectiveness of messages mentioning bene t and personal bene ts of contributing in di erent cohorts. The subjects were assigned to di erent cohorts according to their home country and level of contributions in the past. We reported the results of the execution and replications of this study in an online community. Our original hypothesis that community bene t message would be more e ective in Asian users, and the personal bene t message more e ective in US users was not con rmed (in fact, the observed trend was rather opposite). Moreover, we were not able to register almost no reliable di erences in actions and contributions when dividing students by demography. The only exception is the larger volume of actions pefrormed by Asian students. However, even this observation may not be considered reliable since the overal number of engaged Asian students was low.
At the same time, we discovered that the student status in the system (new or past user) and overal level of activity (active or passive users) appeare to be more reliable factors to predict student behavior. The fact of being a newcomer in the system, having contributed before to the system or being a new student are the most signi cant factors that predict the level of contribution that the messages generated.
While we are still interested to explore the value of demographic factors in personalizing engagement stragegies, we want to shift main focus of our work to adapting the engagement messages to the level of participation in the system. Another venue of research will evaluate the survival rates of the subjects of this study considering factors such as the kind of message they received and their navigation patterns. The ultimate goal is to propose adaptive engagement mechanisms as a way to increase the e ectiveness of the engagement strategies.