In this paper, we describe our end-to-end content-based image retrieval system built upon Elasticsearch, a well-known and popular textual search engine. As far as we know, this is the ifrst time such a system has been implemented in eCommerce, and our eforts have turned out to be highly worthwhile. We end up with a novel and exciting visual search solution that is extremely easy to be deployed, distributed, scaled and monitored in a cost-friendly manner. Moreover, our platform is intrinsically flexible in supporting multimodal searches, where visual and textual information can be jointly leveraged in retrieval. The core idea is to encode image feature vectors into a collection of string tokens in a way such that closer vectors will share more string tokens in common. By doing that, we can utilize Elasticsearch to eficiently retrieve similar images based on similarities within encoded sting tokens. As part of the development, we propose a novel vector to string encoding method, which is shown to substantially outperform the previous ones in terms of both precision and latency. First-hand experiences in implementing this Elasticsearchbased platform are extensively addressed, which should be valuable to practitioners also interested in building visual search engine on top of Elasticsearch.
• Information systems → Image search; • Applied computing → Online shopping; Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). SIGIR 2018 eCom, July 2018, Ann Arbor, Michigan, USA © 2018 Copyright held by the owner/author(s). 1
Elasticsearch [
But a picture is more than often worth a thousand words. With
the explosive usage of phone cameras, content-based image
retrieval [
Finding images relevant with the uploaded picture tends to
be much more involved and vaguer than retrieving documents
matching keywords [
In this paper, we describe our end-to-end visual search platform built upon Elasticsearch. As far as we know, this Image descriptor
Vector to string tokens encoding Database images Indexing Query image
Image descriptor
Indexing
Retrieval based on token matching
Reranking within
is the first attempt to achieve this goal and our eforts turn
out to be quite worthwhile. By taking advantage of the
mature engineering design from Elasticsearch, we end up with
a visual search solution that is extremely easy to be deployed,
distributed, scaled and monitored. Moreover, due to
Elasticsearch’s disk-based (and partially memory cached) inverted
index mechanism, our system is quite cost-efective . In contrast to
many existing systems (using hashing-based [
Since the image preprocessing step and the image feature
extraction step involved in our system are standard and
independent of Elasticsearch, in this paper we address more
towards how we empower Elasticsearch to retrieve close
image feature vectors, i.e., the Elasticsearch-related part of the
visual system. Our nearest neighbor retrieval approach falls
under the general framework recently proposed by Rygl et al.
[
The rest of the paper is organized as follows. In Section 2, we describe the general pipeline of our visual search system, and highlight a number of engineering tweaks we found useful when implementing the system on Elasticsearch. In Section 3 and 4, we focus on how to encode an image feature vector into a collection of string tokens—the most crucial part in setting up the system. In Section 3, we first review the element-wise rounding encoder and address its drawbacks. As a remedy, we propose a new encoding scheme called subvector-wise clustering encoder, which is empirically shown in Section 4 to much outperform the element-wise rounding one. 2
GENERAL FRAMEWORK OF VISUAL SEARCH WITHIN ELASTICSEARCH The whole pipeline of our visual search engine is depicted in Figure 1, which primarily consists of two phases: indexing and searching.
we will first encode them into string tokens
X := {x1, x2, . . . , xn } ⊆ Rd ,
S := {s1, s2, . . . , sn } , where si := E(xi ) for some encoder E(·) converting a ddimensional vector into a collection of string tokens of cardinality m. The original numerical vectors X and encoded tokens S, together with their textual metadata (e.g, product titles, prices, attributes), will be all indexed into the Elasticsearch database, to wait for being searched.
Searching. Conceptually, the search phase consists of two steps: retrieval and reranking. Given a query vector xˆ , we will ifrst encode it into sˆ := E(xˆ ) via the same encoder used in indexing, and retrieve r (r ≪ n) most similar vectors R := xi1 , xi2 , . . . , xir as candidates based on the overlap between the string token set sˆ and the ones in {s1, s2, . . . , sn }, i.e., {i1, i2, . . . , ir } = r-arg max |sˆ ∩ si |.
i ∈ {1,2, ...,n } We will then re-rank vectors in the candidate set R according to their exact Euclidean distances with respect to the query vector xˆ , and choose the top-s (s ≤ r ) ones as the final visual search result to output, i.e.,
s-arg min i ∈ {i1,i2, ...,ir }
∥xi − xˆ ∥2 .
As expected, the choice of E(·) is extremely critical to the success of the above approach. A good encoder E(·) should encourage image feature vectors closer in Euclidean distance to share more string tokens in common, so that the retrieval set R obtained from the optimization problem (2.3) could contain enough meaning candidates to be fed into the exact search in (2.4). We will elaborate and compare diferent choices of encoders in the next two sections (Section 3&4).
Implementation. In this part, we will address how we implement the retrieval and reranking steps in the searching phase eficiently within just one JSON-encoded request body (i.e., JSON 1), which instructs the Elasticsearch server to compute (2.3) and (2.4) and then return the visual search result in a desired order (via Elasticsearch’s RESTful API over HTTP).
For the retrieval piece, we construct a function score query
[
For the reranking piece, our initial trial is to fetch the top-r
image vectors from the retrieval step, and calculate (2.4) to
re-rank them outside Elasticsearch. But this approach
prevents our visual system from being an end-to-end one within
Elasticsearch, and thus makes it hard to leverage many
useful microservices (e.g., pagination) provided by Elasticsearch.
More severely, this vanilla approach introduces substantial
latency in communication as thousands of high-dimensional
and dense image embedding vectors have to be transported
out of Elasticsearch database. As a remedy, we design a query
rescorer [
Multimodal search. More often than not, scenarios more
complicated than visual search will be encountered. For
instance, a customer might be fascinated with the design and
style of an armoire at her friend’s house, but she might want
to change its color to be better aligned with her own home
design or want the price to be within her budget (see Figure
2). Searching using the picture snapped is most likely in vain.
To better enhance customers’ shopping experiences, a visual
search engine should be capable of retrieving results as a joint
outcome by taking both the visual and textual requests from
customers into consideration. Fortunately, our
Elasticsearchbased visual system can immediately achieve this with one
or two lines modifications in JSON 1. In particular, filters can
be inserted within the function score query to search only
among products of customers’ interests (e.g., within certain
price range [
The success of our approach hinges upon the quality of the
encoder E(·), which ideally should encourage closer vectors to
share more sting tokens in common, so that the retrieval set R
found based on token matching contains enough meaningful
candidates. In the following, we first review the element-wise
rounding encoder proposed by Rygl et al. [
Drawbacks. Although the filtering strategy is suggested to
maintain a good balance between feature sparsity and search
quality [
(xˆi − xi )2,
i=1
is summed along each axis equally rather than biasedly based
on the magnitude of xˆi (or xi ). In specific, a mismatch/match
with a (rounded) value 0.01 does not imply that it is less
important than a mismatch/match with a 0.99, in terms of their
contributions to the sum (3.1). What essentially matters is the
deviation ∆ i := xˆi − xi rather than the value of xˆi (or xi ) by
itself. Therefore, entries with small magnitude should not be
considered as less essential and be totally ignored. Second, the
eficacy of the filtering strategy is vulnerable to data
distributions. For example, when the embedding vectors are binary
codes [
In the next subsection, we will propose an alternative encoder, which keeps all value information into consideration and is also more robust with respect to the underlying data distribution. 3.2
Diferent from the element-wise rounding one, an encoder
that operates on a subvector level will be presented in this
part. The idea is also quite natural and straightforward. For
any vector x ∈ Rd , we divide it into m subvectors1,
[x1, . . . , xd/m , xd/m+1, . . . , x2d/m , . . . . . . , xd−m+1, . . . , xm ].
| {xz1 } | {xz2 } | x{zm }
Denote Xi := x1i, x2i, . . . , xni as the collection of the i-th
subvectors from X for i = 1, 2, . . . , m. We will then separately
apply the classical k-means algorithm [
→ {1, 2, . . . , k } assigning each subvector to the cluster index it belongs to. Then for any x ∈ Rd , we will encode it into a collection of m string tokens
“pos1cluster{A1(x 1)}”, “pos2cluster{A2(x 2)}”, . . . . (3.3) The whole idea is illustrated in Figure 3. The trade-of between search latency and quality is well controlled by the parameter m. In specific, a larger m will tend to increase the search quality as well as the search latency, as more string tokens per each vector will be indexed.
(3.2)
In contrast with the element-wise rounding encoder, our subvector-wise clustering encoder obtains m string tokens without throwing away any entry in x , and will generate string tokens more adaptive with the data distribution, as i the assignment function A (·) for each subspace is learned through Xi (or data points sampled from Xi ).
1 2 7 4 3 6 5 1 3 2 5 4 1 2 7 6 5 4 3 In this section, we will compare the performance of the subvectorwise clustering encoder and the element-wise rounding one in terms of both precision and latency, when they are being used in our content-based image retrieval system built upon Elasticsearch.
Settings. Our image datasets consists of around half a
million images selected from Jet.com’s furniture catalog [
Evaluation. To evaluate the two encoding schemes, we
randomly select 1,000 images to act as our visual queries. For
each of the query image, we find the set of its 24 nearest
neighbors in Euclidean distance, which is treated as gold
standard. We use Precision@24 [
Results. In Table 1, we report the Precision@24 and search latency averaged over the 1,000 queries randomly selected. Results corresponding to p ∈ {2, 3} or r ∈ {24, 48} are skipped as they are largely outperformed by other settings. Configurations that can achieve precision ≥ 80% and latency ≤ 0.5s are highlighted in bold. From Table 1, we can see that the subvector-wise encoder outperforms the element-wise one, as for all results obtained by the element-wise encoder, we can find a better result from the subvector-wise one in both precision and latency. To better visualize this fact, we plot the Pareto frontier curve over the space of precision and latency in Figure 4. In specific, the dashed (resp. solid) curve in Figure 4 is plotted as the best average Precision@24 achieved among all configurations we experiment for element-wise rounding (resp. subvector-wise clustering) encoder, under diferent latency constraints. From Figure 4, we can more clearly observe that the subvector-wise encoder surpasses the element-wise one. Notably, when we require the search latency to be smaller than 0.3 second, the subvector-wise encoder is able to achieve an average Precision@24 as 92.14%, yielding an improvement of more than 11% over the best average Precision@24 that can be obtained by the element-wise one.
Although our subvector-wise clustering encoder outperforms
the element-wise rounding one, it might be still restrictive to
enforce a vector to be divided into subvectors exclusively using
(3.2), which could potentially downgrade the performance of
the encoder. Our next step is to preprocess the data (e.g.,
transform the data through some linear operation x 7→ T [x ] with
T [·] learned from the data) before applying our
subvectorwise clustering encoder. We believe this flexibility will make
our encoding scheme more robust and adaptive with respect
to diferent image feature vectors extracted from various
image descriptors. Another interesting research direction is to
evaluate the performances of diferent encoding schemes in
other information retrieval contexts–e.g., neural ranking model
based textual searches [
We are grateful to three anonymous reviewers for their helpful suggestions and comments that substantially improve the paper. We would also like to thank Eliot P. Brenner and Aliasgar Kutiyanawala for proofreading the first draft of the paper. [18] T. Ge, K. He, Q. Ke, and J. Sun. 2013. Optimized product quantization for approximate nearest neighbor search. In Proceedings of CVPR. 2946–2953. [19] C. Gennaro, G. Amato, P. Bolettieri, and P. Savino. 2010. An approach to content-based image retrieval based on the Lucene search engine library.
In Proceedings of TPDL. 55–66. [20] Y. Gong, S. Lazebnik, A. Gordo, and F. Perronnin. 2013. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence