Search engines are the most utilized tools to access information on the Web, and they are widely used to support learning activities, even though they are specially used for acquiring factual knowledge, and do not support searching as learning tasks. In this paper we present SaR-Web, a search tool that allows the visualization of search results with a semantic added value in order to facilitate comparisons and further analysis. SaR-Web was originally developed to support search as research activities: it allows to analyze search results under diverse cultural and social perspectives. By fostering critical thinking and creative learning, SaR-Web promotes search as learning processes. A recent evaluation of the system by students attending the WebScience course at the University of Hanover is presented. Preliminary results highlight the positive impact that the system had on students in stimulating creativity and critical thinking.
Search engines are the most utilized tools to access information on
the Web, and their impact on the learning processes occurring
during search sessions has been investigated by several authors
[
These studies have highlighted that typical search systems may
work well for certain kinds of basic search tasks, but do not support
more complex search tasks such as searching to learn and searching
to investigate [5]. In other words, as Rieh et al. suggest [
Search engines do not attempt to promote and encourage reflection on engine workings as well as outputs in any systematic way, e.g., by encouraging comparison and reflection on search results for the same query across different language domains.
Even though some automatic tools are available to support analytical and reflective practices to discover new knowledge or information, they are mainly limited to specific sub-activities or focus on specific knowledge repositories, such as Wikipedia. Consequently, such practices require manual work to prepare the search engine, specify local-domain settings for the country specificity of the languages, rank lists of results and refine the query.
For this reason we developed a web search tool, named SaR-Web
(Search as Research-Web), to support the investigation of broader
research questions [
The relationships between the concept of search as research and
the searching as learning one is even deeper: both share the same
vision of the search process as a mechanism to develop new
knowledge, and proceed from a set of epistemic beliefs that
emphasize knowledge as constructed via diverse perspectives [
Inspired by the (largely manual) work of search engine comparison
by the Digital Methods Initiative, we developed a web search tool
SaR-Web that allows the visualization of search results with a
semantic added value in order to facilitate comparisons and further
analysis [
SaR-Web adopts an automatic method using the Bing API to collect search results for a specific query and put forward a semantic approach to compare results in four different languages, providing visual representations of search results, without the necessity to understand fluently all languages under investigation. We use Bing as a search engine because the Bing API is available for research (while Google, under its terms of service, is not).
The comparison of search results in different languages is often hindered by the difficulties in performing traditional textual analysis. To this end, in SaR-Web we implement a semantic based approach in which the comparison between search results in different languages is supported through visualization of semantic concepts, thereby overcoming the limit of textual descriptions. Moreover, SaR-Web annotates the query results semantically, so that the user (researcher or student) analyses and compares results on the semantic level, and not only on the syntactic one, thus facilitating high level learning processes, in terms of Bloom’s taxonomy.
SaR-Web provides word clouds in four languages (English, German, French and Italian) which highlight the most relevant keywords in localized Web sites. From a technical perspective SaRWeb uses the Dandelion’s Entity Extraction API1 to generate (semantic) word clouds in the respective language. The workflow of the system is as follows. After a user searches for a keyword, the returned results (URLs) are obtained from the Bing search and are sent to Dandelion’s Entity Extraction API. The Entity Extraction API parses the content and sends back a response containing the extracted Wikipedia entities along with other information. The Wikipedia entities from the response are used to extract the DBpedia2 concepts that are in turn filtered and indexed for that specific search. The frequencies of the retrieved concepts are used to visualize the word cloud.
In detail, the main tasks performed by SaR-Web are:
Localized search: the keywords introduced by the user are searched by using specific language and local settings (e.g., “language:it loc:it”) so that only web pages from a specific country or region, and written in a specific language, are returned.
Named entity recognition: the title and the snippet text from the body retrieved from search engine results are elaborated with the Dandelion NER (Named Entity Recognition) service. This service returns the Wikipedia reference extracted by the NER procedure. This operation is performed for the four languages supported by SaR-Web: English, Italian, German, and French.
Semantic annotation: SaR-Web transforms the Wikipedia reference in each language to the correspondent concept in the DBpedia knowledge base.
Visualization: a tag cloud is generated with the main concepts (or keywords) for the four supported languages (Fig. 1) SaR-Web also supports in-depth investigation by displaying the search results related to a specific concept shown in the tag cloud, when the user clicks on it. 1 https://dandelion.eu/docs/api/datatxt/nex/v1/
1) To what extent is it possible to automatize the query analysis to support search as research? We want to automatize the analysis because: (a) Manual investigation is time consuming, (b) Web search results vary over time and require a big effort to replicate the analysis manually (c) Better visualization of results can be provided. 2)
To what extent automatic extraction and visualization of concepts help students in activating critical thinking and supporting search as learning processes? We start from the assumption that search as research methods are applicable to the educational context and stimulate students’ creativity.
We presented our research study and the SaR-Web system to the students attending the WebScience course at the University of Hanover, Germany, during the Summer Semester 2016. A group of 15 students participated to the experimentation.
Most students had a very high proficiency in English and some of them in German; none of them knew Italian or French. For this reason, we provided a simplified system interface that only displays English and German results.
In order to investigate the first research question we provided students with specific accounts according to the type of evaluation (manual or automatic), and we prepared two different views of the search results. Students who could easily read search results in the original language (9) carried out the manual evaluation, the remaining 6 students who could not understand German did the automatic evaluation analyzing the concepts provided in English by SaR-Web.
For the manual evaluation, students received a list of the first 10 results returned directly from the Bing API (to avoid any bias introduced by personalization settings) in the two languages English and German.
For the automatic evaluation we (a) set up two predefined queries “rights” and “nuclear” so that the students did not have to type them and translate them into German, (b) set fixed searching, parameters in order to optimize the results according to the number of concepts displayed in the tag cloud, (c) provided a list of weighted concepts to better understand the concepts in the tag cloud, (d) associated colours in the word cloud according to categories and concepts, in order to highlight them across the results in different languages.
Manual evaluation - Students accessed the SaR-Web interface providing a ranked list of 10 results retrieved through the Bing API. Task 1: Skim through the content of each page (full-text) and collect the concepts that are most relevant to the query. Briefly comment the page and list the main concepts.
Task 2: Briefly comment the results (e.g. are the results diverse enough? Are they comprehensive to represent the concept? Did you expect different results?) 2 DBpedia is the semantic knowledge base extracted from Wikipedia http://dbpedia.org/ Task 3: Compare the results between the two markets3 (US and De) Automatic evaluation - Students accessed the SaR-Web interface providing two search fields for the two languages. Each search returned the tag cloud visualizing the most popular concepts related to the query in both languages, including a weighted list to indicate the relevance of each concept in the word cloud.
Task 1: Explore/analyze the tag clouds generated in SaR-Web for English and for German and discuss the most popular concepts for each local domain. A list of ranked results is also provided by the system.
Task 2: Compare the results between the two markets and comment them (e.g. do you find specific concepts which appear only in one country and not in the other? Do you have a possible explanation for that?).
Task 3: Open the source web pages (click on the concepts in the tag cloud) and compare the previous results with the information you get looking at the full-text.
Students had 1 hour to complete the evaluation.
At the end, each student wrote a summary comparing the results and sent the document (e.g. Word file) via e-mail to one of the authors.
We are currently collecting the evaluation reports and start analyzing the results. 3 Language and country/region information according to Bing API terminology Preliminary feedback received during the lesson was encouraging as students were interested in the research approach and in exploring the SaR-Web system also beyond the specific experiment: they asked if they could try queries of their choice to see the most popular concepts and how results differ in both markets.
The qualitative analysis of the reports has highlighted how the use of the system has fostered students’ critical thinking processes, although similarities and differences in the two groups have emerged.
As regards the comparison of search results, in general students confirmed the difference of most popular concepts in the two markets. For example student webscience01analysing the query “nuclear” with the automatic evaluation noticed that “The first 10 results in the English and in the German version provide different concepts. In English we see that most popular concepts are about the danger from Nuclear technologies (different types of weapons and disasters). And in German it is more about medical usage of nuclear science”.
With respect to the task 3 of the Automatic evaluation, by looking at the full text of the whole page, some students confirmed the same concepts provided by the system, other students found different concepts. For example student webscience01 wrote: “In comparison to the SaR-Web tag clouds, if I look at the whole page I find the same concepts (checked Human rights and Rights, Nuclear power, Nuclear weapon and NRC: Nuclear Materials)”. Student webscience02: [query rights] “In comparison to the SaRWeb tag clouds, if I look at the whole page I find very different concepts”; [query nuclear] “if I look at the whole page I find the same concepts”.
The manual evaluation gave similar results as the automatic one: in both cases students identified differences in pages from different domains. For example student webscience07 in the manual evaluation for the query “nuclear” pointed out that “The Wikipedia pages in German are completely different from the Wikipedia pages in English, the former being about Nuclear Medicine and Nuclear Spaces (related to Mathematics)”.
One student focused on term disambiguation noticing that “in every retrieved result there were articles from the (local) Wikipedia contained in the top positions. They provided a good disambiguation, along with a huge amount of detailed information about the different aspects of meaning”.
In general, we notice evidence of the positive impact that the system had on students in activating critical thinking. Students were curious to investigate other "research questions", extract new knowledge and learn new concepts from the comparison of results in various languages. 5. DISCUSSION AND CONCLUSIONS Moving from the manual inspection of engine results to automatic keyword extraction is challenging in that, both certain concepts in the database, as well as those missing in the database would have been eliminated in an editorial approach. On the other hand, supporting search as research through automated means clearly makes it easier to analyze more queries and results, with the possibility of manual inspections to uncover additional insights. Thus the automated search as research work both should strive to continue to perfect the outputs, as well as being used as an intermediary step, prior to an editorial polishing. Finding solutions to address this fascinating and still open challenge requires the contribution of experts from different research fields and expertise such as computer scientists, sociologists and digital humanities experts.
SaR-Web aims at supporting searching as learning processes.
Following Rieh et al. ([
SaR-Web has the capacity to log the users’ interactions within the search environment, thus providing users’ behavior data which can be used to analyse the learning process during the search sessions.
SaR-Web allows to compare different local domains it supports critical and creative learning processes which are central to SaR and SAL.
SaR-Web can impact on exploratory web searches, by reducing the time spent by students in identifying multiple aspects of a specific topic.
To sum up, SaR-Web has the potential to support high-level learning activities described in Bloom’s taxonomy such as identifying and analyzing patterns (amongst results provided by search engines under specific cultural and linguistic requirements), comparing different ideas (emerging from different local versions of search engines), integrating ideas, and creating new ideas (as the outcome of the research task).
Our thanks to all students attending the WebScience course who participated in the evaluation of the system. This work was partially funded by the European commission in the context of the ALEXANDRIA project (ERC advanced grant no 339233).