As Udi Manber, currently a VP of Engineering at Google, likes to say: "Search is hard!" Yet, it is possible to build a rather powerful search engine that provides rather accurate answers most of the time. In the talk, I brie y discuss some of the fundamental concepts and technologies behind search engines, starting with a quick review of Information Retrieval and then moving through 15 years of search engines evolution. Following, I discuss some of the current challenges related to the search task, as well as open opportunities for research.
breaks down with larger collections. To deal with large collections, such as the ones dealt with by modern search engines, the retrieval model needed to evolve.
In the early days of the Web, search engines were implemented as classic
IR systems. Their rst big evolutionary step was to recognize that the links
among Web pages provide additional information that is of value for ranking.
Indeed, a Web page that is pointed to by many links re ects a degree of authority
on a given topic of interest, a degree of authority that can be computed as a
normalized count of the number of inlinks. Thus, when a user poses a query,
the engine can return rst the pages that match the query and have the highest
degrees of authority. This is the idea behind the now famous Page Rank, the
ranking function used to build Google [
Since then, search engines have evolved to include into the ranking function a variety of signals originary from the document collection, from the query stream, and from the user actions. As a result, modern search engines now combine hundreds of signals into a single ranking function in an attempt to better answer the user queries. During the talk, I will discuss some of these signals and how they are used for ranking purposes. 2
Challenges and Opportunities While search engines have evolved continuously over the last 15 years, they still face many challenges, particularly with infrequent and detailed queries. A user that is seeking a phone number of their doctor will frequently be frustrated with the answers produced by the search engine. To cope with queries of this nature, search engines need to evolve further. They need to evolve to incorporate knowledge encoded in some form that it can be useful for ranking purposes.
Also important, it is frequently the case that determining the most relevant document for a query requires interpreting the content of the document and determining its central topics. That is, document understanding is an area little understood and which needs much more research, if search is to be further improved. During the talk, I will discuss additional examples of challenges and opportunities in the space of search.