Text representations are widely used in NLP tasks such as text classification. Very powerful models have emerged and been trained on huge corpora for diferent languages. However, most of the pre-trained models are domain-agnostic and fail on domain-specific data. We perform a comparison of diferent text representations applied to French Real Estate classified advertisements through several text classification tasks to retrieve some key attributes of a property. Our results demonstrate the limitations of pre-trained models on domain-specific data and small corpora, but also the strength of text representation, in general, to capture underlying knowledge about language and stylistic specificities.
Real-estate classified advertisements provide great details and relevant information about a property that is valuable for the intelligence of the real-estate market. For example, price predictions are often based on attributes such as the type of property, the number of rooms or even the floor at which it is located. However, those key information are not always clearly specified in the ads and often difer from an ad to another and from an advertiser from another. Thus, the automatic extraction of key information from the text of real-estate classified advertisements is a challenging and promising task to help in some real-estate market applications.
Given the limited number of diferent values that economically relevant attributes of a real-estate property may take, this information extraction task can be viewed as a classification problem. For instance, if the number of rooms of a property is sought for, the problem can be stated as assigning the property described by a given advertisement to one of the classes labeled as {1, 2, 3, 4, 5+}, corresponding to single-room, tworoom, three-room, four-room, and five-or-more-room dwellings. Nevertheless, texts of real-estate ads are often short with language and stylistic specificities and variabilities. Also, an additional challenge may be represented by the need to extract information from advertisements written in languages other than English, for which linguistic resources are thus harder to find or less well-developed.
Text representations models have emerged as very powerful approaches to learn useful features of a text and have been widely used for Machine Learning tasks such as classification. It is thus interesting to carry out a comparative study of the most prominent models found in the literature as they are applied to this specific text classification task, to understand their strengths and weaknesses.
Our main contributions may be summarized as follows: • we apply diferent text representations to a classification task to retrieve key attributes of properties found in classified advertisements written in French, that we have collected and annotated manually; • we propose a comparison of the strength and limitations of diferent text representations ; • we analyse the vocabulary and register used by French real-estate agents to understand their impact in the classification.
positions our contribution with respect to the literature; Section 3 provides a detailed description of the dataset and the method we propose to classify real-estate ads; Section 4 reports the results of the experiment and draws some conclusions. into numeric features.
words (BoW), which represents a text by the occurrence of words. This model involves a vocabulary and a measure of the occurrence of words. The vocabulary captures all the words found in the corpus of texts and is fixed.
Regarding the measure, it can be binary (presence or not
of a word in the text) or weighted. For example, Term
Frequency – Inverse Document Frequency (TF-IDF) is
a weighted BoW that scores each word of a text by its
frequency in the text (TF) and across the whole corpus
(IDF). This measure penalizes very frequent words and Figure 1: CBOW and Skip-gram architectures [
Those limitations led to a feature learning technique representation of words. Nevertheless, those kinds of
in which words are mapped to a vector of dimensions, modes do not improve performance significantly and are
with smaller than the size of the vocabulary. This ap- computationally very expensive.
proach is called Word Embedding. The most well-known The last state-of-the-art context-based model uses a
model is Word2Vec, developed by Mikolov et al. [
A variant of Word2Vec is Doc2vec [
Finally, there exist other models similar to Word2Vec, training set and a pre-trained model on general domain
such as Global Vectors (GloVe) [
All of the feature learning techniques presented above All the methods described above have been widely have limitations such as the need for a huge corpus to used and compared for diferent tasks and types of text. train and failure to capture contextual information. However, a few of them are focused on small data and
Context-based models try to tackle those limitations. French documents, as in our work.
Contextual embeddings help to distinguish a same word Dynomant et al. [
Another work [
A recent survey of text classification algorithms [
Finally, Gupta and Waseem [
Tables 1, 2, and 3 show the class distribution for each target. The type of property is complete (0 missing data) 3. Method but not really balanced. It is very easy to get the type of property in the metadata since it is essential to sell the In this section, we will first describe the dataset, then the product. However, the number of rooms and the floor of preprocessing and feature extraction applied on the ad- a property are not complete. This is the reason why it
French real-estate ads are full of noisy, repetitive words and abbreviations.
First, we removed noise such as elongated punctuations (“......”, “!!!!”, etc.) and URLs. We replaced symbols such as “e” by “euros”, “m2”, “M2”, and “M2” by “m2”. Also, some abbreviations are of common use in the real-estate market, e.g., ‘apt.’ stands for apartment, ‘balc.’ for balcony.
We tried to remove proper nouns found in phrases such as “contactez Paul Martin” (contact Paul Martin), which refer to the advertiser, thanks to a regex.
Then, we lemmatized texts with a French lemmatizer from spaCy. This lemmatizer has been trained on the French Sequoia corpus1 and WikiNER. The lemmatizer performs pretty well for French, as we can see in Figure 3.
Nevertheless, the syntax of real-estate ads is slightly different from the general French syntax and the lemmatizer fails to assign the correct lemma. For example, the noun “nuit” (night) in the phrase “coin nuit” (sleeping area) is erroneously lemmatized in the verb “nuire” (nuit is also the 3rd person sg. form of “nuire”, to harm).
The final step was to remove punctuation, numbers and French stopwords. We tuned the stopword list with very frequent words in the real estate vocabulary, e.g., “honoraires” (fees), “agent immobilier” or “agent commercial” (real-estate agent). might be interesting to predict this kind of information 3.3. Feature Extraction from the text. Also, we only predict the number of rooms for apartments and houses, and the floor of the property After cleaning the texts, we applied diferent text repreonly for apartments. sentations methods presented in Section 2.
Although we plan on making our dataset publicly avail- Two classical Bag-of-Words methods have been tested
able, for the time being we could not, for legal reasons, : TF and TF-IDF. We set maximum features to 5,000 and
as it is expressly forbidden by the real-estate agencies we chose to take a range of -grams from 1 to 3, that is
that own those advertisements to republish them in any to say from 1 to 3 tokens.
form, as long as the relevant properties are still on the Then, we tried non-contextual embedding with
market. Word2Vec (Skip-Gram), Doc2Vec (PV-DM), and FastText.
We trained those 3 models on our corpus of ads. We
chose a smaller number of features for FastText because
3.2. Text Preprocessing of its need of memory storage. We wanted to compare
our own trained non-contextual embeddings with a
pretrained model, but only few models have been trained
on a French corpora. Thus we only found a pre-trained
Word2Vec model [
Finally, we wanted to apply contextual embedding.
However, although pre-trained (Bi-)LSTM models such
as ELMo have recently become available for French,2 we
were not aware of them at the time we planned our
experiments. Now, it is very long to train a model and a
huge corpus is needed, which we lack. Therefore, we
decided to try pre-trained Transformer models for French,
like CamemBERT [
ing to three labels (Type of property, Number of rooms
ELMoForManyLangs. https://github.com/HIT-SCIR/ and Floor of the property) in order to retrieve missing metadata. We used our diferent feature extractions as input and we applied a classifier. Diferent classifiers have been tried (e.g. Naive Bayes, Logistic Regression, or Random Forest) for every text representations, but in the following, we will only present results with Logistic Regression as it gave higher score. CamemBERT and FlauBERT have been already trained for classification, so we used their classifier based on neural networks.
We also compared our models to a simple Regex, which is our baseline. For instance, to find the type of property, we crafted rules that classify a property as an apartment if the words “apartment” or “studio” are found; else if “house” or “villa” are found, it is classified as house, and so on. For the number of rooms, we searched for a number (in words or digits) before the word “pièce” (room) or “chambre” (bedroom); if we find a number before “pièce” then we take it as the label for classification; else, if we ifnd a number before “chambre”, we take that number plus one. The same idea is followed for the floor of a property, but the word “étage” (floor) is used instead.
We used the F1-Score as measure of accuracy to handle imbalanced classes.
periments and the comparison of the diferent text representations. Afterward, we will discuss the possible explanations of the results and infer knowledge for the real-estate market.
text representation methods combined with a classifier outperform the Regex baseline. It shows that Real Estate ads are too noisy to retrieve information easily. Nevertheless, we noticed that the predictions of number of rooms are slightly better with a regex for 2, 3 or 4 rooms. The advertisers seem to write more often the number of rooms for those classes than for ’1’ or ’5 and more’ rooms in their description.
Comparing text representations, classical methods (mainly TF-IDF) and CamemBERT achieved the best F1-Score. Non-contextual methods such as Word2Vec, Doc2Vec or FastText lagged behind. FlauBERT fared even worse. FastText and FlauBERT are more based on a character or sub-word level. However, the vocabulary of French real-estate ads is pretty poor and might not be suitable for this level. On the contrary, Doc2Vec embeds the whole paragraph and is less precise. Most paragraphs contain the same kind of vocabulary and syntax, so Doc2Vec fails to discriminate them. In general, classical methods gave better results, except for CamemBERT, which has similar results. Nevertheless, the training dataset was really small and the performance of more complex methods might improve with a larger corpus.
results. Odds ratio helps to know if a variable (e.g., a
word) is increasing or decreasing the probability of the
text belonging to one class or another. In Tables 7, 8
and 9, we can see the words that increase or decrease
the most the probability for each class. For example, the
probability is obviously increased by the word that
describes the class, such as “apartment”, “house”, for the
type of product, or “two rooms”, “three rooms” for the
number of rooms. However, it is interesting to see that stand out for the classification task.
unexpected words turn out to have an important impact. Another interesting finding is that pre-trained models,
Indeed, we can see that real-estate agents target diferent despite having been trained on a much larger corpus and
people according to the number of rooms. They often use with impressive computational resources, turn out not
words about investment for one- or two-room dwellings, to be much useful, due to their being domain-agnostic,
while they target parents with children for three-room whereas models trained on our small domain-specific
properties. Furthermore, the vocabulary used to clas- corpus clearly outperform them, thus confirming similar
sify the floor of the property is also diferent. An agent ifndings in other domains [
In a nutshell, this study points out the specific and We believe that our results will provide useful guidance stylistic vocabulary used by French real-estate agents. As to anybody willing to engage in real-estate classified ads we have shown, basic information is not always clearly classification, in general, and information extraction in and explicitly written in the ads. However, other words particular. can help to infer such key information for the real-estate A promising research direction which is, in our opinmarket. ion, worth investigating is to combine diferent text representations models, each capable of capturing diferent details, in order to obtain a higher overall accuracy. An5. Conclusion and Further Work other quite obvious extension of our investigation would be to gather a bigger corpus of advertisements and study how using it to train the model increases the classification accuracy. Large language models could also be considered, as a possible end-to-end solution to this information extraction task.
As future work, it could be interesting to capture the syntax of real-estate ads in order to understand even better their language. Also, property pictures, which of
representations models with respect to their application to the classification of real-estate classified advertisements in French, having the ultimate goal of extracting key information about the properties they advertise.
In summary, we found out that, among the models we tested, the classic representation TF-IDF and the most state-of-the-art model CamemBERT are the ones that ten come together with the advertisements, are a major source of information that can contribute to the intelligence of the real-estate market.
which is a joint research team of Université Côte d’Azur, Inria, and I3S. Our research motto: AI in bridging social semantics and formal semantics on the Web.
This work has been partially supported by the French government, through the 3IA Côte d’Azur “Investments in the Future” project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA0002.