We present an initial examination on the impact text complexity has when incorporated into the recommendation process in community question answering sites. We use Read2Vec, a readability assessment tool designed to measure the readability level of short documents, to inform a traditional content-based recommendation strategy. The results highlight the benefits of incorporating readability information in this process.
• Information systems → Social recommendation; Question answering;
Community question answering (CQA) sites allow users to
submit questions on various domains so that they can be answered
by the community. Sites like Yahoo! Answers, StackExchange, or
StackOverflow, are becoming increasingly popular, with thousands
of new questions posted daily. One of the main concerns of such
sites, however, is the amount of time a user has to wait before
his question is answered. For this reason, CQA sites depend upon
knowledge already available and refer users to older answers, i.e.,
answers provided for previously-posted questions and archived on
the site, so that users can get a more immediate response to their
inquiries. This recommendation process has been extensively studied
by researchers using a wide range of content similarity measures
that go from the basic bag-of-words model to semantically related
models, such as ranksLDA [
We argue that the recommendation process within CQA sites
need to go beyond content matching and answer-feature analysis
and consider that not every user has similar capabilities, in terms of
both reading skills and domain expertise. User’s reading skills can
be measured by readability, which refers to the ease with which
a reader can comprehend a given text. This information has been
applied in the past with great success for informing tasks such as
K-12 book recommendation [
In this paper, we present an initial analysis that explores the influence of incorporating reading level information into the CQA recommendation process. With this objective in mind, we consider the answer recommendation task, where a user generates a query that needs to be matched with an existing question and its corresponding answer. We address this task by ranking question-answer pairs and selecting the top-ranked pair to recommend to the user. For doing so, we build upon a basic content-based recommendation strategy which we enhance using readability estimations. Using a recent Yahoo! Question-Answering dataset, we measure the performance of the basic recommender and the one informed by text complexity and demonstrate that readability has indeed an impact on user satisfaction. 2
We describe below the strategy we use for conducting our analysis. Given a query q, generated by a user U , we locate each candidate answer Ca —along with the question Qa associated with Ca —that potentially addresses the needs of U expressed in q. Thereafter, the highest-ranked Ca -Qa pair is recommended to U . 2.1
To perform content matching, we use an existing WordNet-based
semantic similarity algorithm described by Li et al. in [
To estimate the reading level of Ca and U (the latter inferred
indirectly through q), we first considered traditional readability
formulas, such as Flesch Kincaid [
Given that the answer to be recommended to U should match U ’s reading ability to ensure comprehension, we compute the Euclidean distance between the corresponding estimations, using Equation 1.
d(q, Ca ) = R2V (q) − R2V (Ca) (1) where R2V (q) and R2V (Ca) are the readability level of q and Ca , respectively, estimated using Read2Vec. 2.3
We use a linear regression model1 for combining the scores computed for each Ca –Qa pair. This yields a score, Rel (Ca , Qa ), which we use for ranking purposes i.e., the pair with the highest score is the one recommended to U .
Rel (Ca , Qa ) = β0 + β1Sim(q, Ca ) + β2Sim(q, Qa ) + β3d(q, Ca ) (2)
where β0 is the bias weight, and β1, β2 and β3 are the weights that
capture the importance of the data points defined in Sections 2.1 and
2.2. This model was trained using least squares for optimization.
For analysis purposes, we use the L16 Yahoo! Answers Query to
Questions dataset [
In addition to our Similarity+Readability recommendation strategy (presented in Section 2), we consider two baselines: Random, which recommends question-answer pairs for each test query in an arbitrary manner; and Similarity, which recommends questionanswer pairs for each test query based on the content similarity between the answer and the query, computed as in Section 2.1.
An initial experiment revealed that regardless of the metric, i.e., Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG), the strategies exhibit similar behavior, thus we report our results using MRR.
As shown in Figure 1, recommendations generated using the semantic similarity strategy discussed in Section 2.1 yield a higher MRR than the one computed for the random strategy. This is anticipated, as Similarity explicitly captures the query-question and query-answer closeness. More importantly, as depicted in Figure 1, integrating readability with a content-based approach for suggesting question-answer pairs in the CQA domain is efective, in terms of enhancing the overall recommendation process2. In fact, as per its reported MRR, Similarity+Readability positions suitable question-answer pairs high in the recommendation list, which is a non-trivial task, given that for the majority of the test queries (i.e., 83 %), there are between 5 and 23 candidate question-answer pairs. 1We empirically verified that among well-known learning models, the one based on linear regression was the best suited to our task. We attribute this to its simplicity, which can better generalize over few training instances than most sophisticated models. 2The weights learned by the model: β0, β1, β2, β3 = {2.26, 0.58, 0.20, 0.12} .
In this study, we analyzed the importance of incorporating readability level information into the recommendation process when it comes to the community based question answering domain. We treat the reading level as a personalization value and compare the readability level on an answer with respect to the reading abilities of a user, inferred through his query. We demonstrated that reading level can be an influential factor in terms of deciding the answer quality and can be used to improve user satisfaction in a recommendation process.
In the future, we plan to conduct a deeper study using other community question answering sites such as Quora or StackExchange. We also plan to analyze queries for additional factors, such as relative content-area expertise, to better predict a user’s familiarity with content-specific vocabulary used on archived answers to be recommended. We suspect that readability and domain-knowledge expertise will be highly influential when the recommendation occurs on CQA sites like StackExchange, given the educational orientation of questions posted on the site.
This work has been partially funded by NSF Award 1565937.