In this paper we propose a methodology to learn to automatically annotate domain-speci¯c information from large repositories (e.g. Web sites) with minimum user intervention. The methodology is based on a combination of information extraction, information integration and machine learning techniques. Learning is seeded by extracting information from structured sources (e.g. databases and digital libraries). Retrieved information is then used to partially annotate documents. These annotated documents are used to bootstrap learning for simple Information Extraction (IE) methodologies, which in turn will produce more annotations used to annotate more documents. It will be used to train more complex IE engines and the cycle will keep on repeating itself until the required information is obtained. The user intervention is limited to providing an initial URL and to correct information if it is the case when the computation is ¯nished. The revised annotation can then be reused to provide further training and therefore getting more information and/or more precision.
The Semantic Web (SW) needs semantically-based
document annotation1 to both enable better document retrieval
and empower semantically-aware agents. Most of the
current technology is based on human centered annotation, very
often completely manual [
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
Copyright 2000 ACM 0-89791-88-6/97/05 ..$5.00 irrelevant for some use(r)s: a page in a pet shop web site can be annotated with shop-related annotations, but some users would rather prefer to ¯nd annotations related to animals.
Di®erent annotation can be imposed on a document using di®erent ontologies. An ontology is required because it describes concepts and relationships that occur in a very restricted view of the real world, basically it describes the domain in which we are working. Most of the annotation is likely to be associated in the future by Web actors other than the page's owner, exactly like nowadays' search engines produce indexes without modifying the page code. Producing methodologies for automatic annotation of pages with no or minimal user intervention becomes therefore important: the initial annotation associated to the document loses its importance because at any time it is possible to automatically (re)annotate the document and to store the annotation in a separate database or ontology. In the future Semantic Web, automatic annotation systems might become as important as indexing systems are nowadays for search engines.
Automatic annotation methodologies have been developed
in the past at the intersection of research areas such as
Information Extraction from text (IE)[
In this paper we propose a methodology to learn how to annotate semantically-consistent portions of the Web extracting and integrating information from di®erent sources. All the annotation is produced automatically with no user intervention apart some corrections the users might want to perform. The methodology has been fully implemented in Armadillo2, a system for unsupervised information extraction and integration from large collections of documents. The natural application of such methodology is the Web, but large companies' information systems are also an option. In this paper we will focus on the Web, and in particular in mining web sites of Computer Science Departments. All the process is based on integrating information from different sources in order to provide some seed annotations. This will then bootstrap learning which in turn will provide more annotations and so on. In synthesis we start with a simple methodology which requires limited annotation, and move on to produce further annotation to train more complex modules.
In the next section we will present the generic architecture that we have used to build the application. Then we will describe the CS Department task and how the information from di®erent sources is integrated in order to learn to annotate the desired information. Experimental results are illustrated and ¯nally we will discuss some future challenges that our experience highlights.
Since there can be many potential applications of the
proposed technology we decided to make the Architecture as
generic as possible. In order to do so, the Architecture had
to be portable and scalable. Portability was achieved by
making use of a simple methodology. The system starts from
a set of generic strategies de¯ning where to search
information and what to look for. When data is harvested using
those strategies, it is passed to an oracle in order to verify
whether it is valid information or not. By oracle we mean a
human or a resource (such as an IE engine, a database etc.)
that can identify which items on a web page are instances of
concepts found in a preloaded ontology. Once we have some
information, the system applies further strategies which can
be used with that kind of information and the results are
given to other more specialised oracles. Basically the
system keeps on looping until there is no more information to
discover or the user decides to interrupt the cycle. It is clear
that we don't make any assumptions on any domain. The
strategies range from simple/generic ones (like looking for
capitalised words) to more complex ones (like setting up and
using IE tools) and they can be easily created for any
domain. This technique works because a key feature of the web
is the Redundancy of information. Redundancy is given by
the presence of multiple citations of the same information in
di®erent contexts and in di®erent super¯cial formats. This
factor is currently used for improving question answering
systems [
To make system scalable we have implemented an architecture based on Web Services where each task is divided in subtasks. Each subtask is performed by a server which in turn will use other servers for implementing parts of the subtask. Each server exposes a declaration of input and output, plus a set of working parameters. Servers are reusable in different contexts and applications. For example one server in the CS department task will return all papers written by a person by accessing Citeseer. Another one will do the same on another digital library . The named entity recogniser server (whose role is to decide if a string is a name) will invoke these servers and integrate the evidence returned and decide if such evidence is enough to conclude that the candidate string represents a person.
Facilities for de¯ning wrappers are provided in our
architecture by Amilcare (nlp.shef.ac.uk/amilcare/), an adaptive
IE system based on a wrapper induction methodology able
to cope with a whole range of documents from rigidly
structured documents to free texts [
The application used to describe the methodology is mining websites of Computer Science Departments. The goal is to discover who works in a speci¯c department (name, position, home page, email address, telephone number) and to extract for each person some personal data and a list of published papers larger than the one provided by services such as Citeseer. The structure and layout of the web pages is di®erent for every department but we can expect to ¯nd some common elements in Computer Science Departments web pages such as a people's page containing the list of people working in the department and many others. 3.0.1
The goal of this task is to discover the names of the people who work in the speci¯c department. This task is more complex than a generic Named Entity Recognition because many irrelevant people's names are cited in a site, e.g. names of undergraduate students, clerics, secretaries, etc, as well as names of researchers from other sites that e.g. participate in common projects or have co-authored papers with members of sta®. Organizing the extraction around a generic Named Entity Recognizer (NER) such as Annie (www.gate.ac.uk) is the most natural option. This does not ¯nish the job, though, because a NER recognizes ALL the people's names in the site, without discriminating between relevant and irrelevant. Moreover classic NER tend to be quite slow if launched on large sites (e.g. the 1,600 pages of the CS department at the University of Southampton) and can be quite imprecise on Web pages, as they are generally de¯ned for newspaper-like articles. A two-step strategy is used here instead: initially a short list of highly reliable seed names are found. Then these seeds are used to bootstrap learning for ¯nding further names. 3.0.2
To ¯nd seed names, a number of weak strategies are combined that integrate information from di®erent sources. First all the web site is crawled looking for strings that are potential names of people (e.g. using a gazetteer of ¯rst names and a regular expression such as <¯rst-name>+(capitalized word)+.). Then the following web services are queried: ² Citeseer (www.citeseer.com): { Input: the potential name; { Output: a list of papers and a URL for home page (if any); ² The CS bibliography at Unitrier (http://www.informatik.uni-trier.de/ ley/db/): { Input: the potential name: { Output: a list of papers (if any); ² HomePageSearch (http://hpsearch.uni-trier.de/): { Input: the potential name; { Output: a URL for home page (if any); ² Annie
(www.gate.ac.uk):
² Google
(www.google.co.uk)
{ Input: the potential name and the text
surrounding it;
{ Output: True/False;
{ Input: the potential name and the URL of the
site in order to restrict search;
{ Output: Relevant Pages that are hopefully home
pages;
The information returned by the digital libraries (Citeseer
and Unitrier) is used to con¯rm or deny the name identity
of the string. If they return reasonable results for a
speci¯c name (i.e. not too few and not too many), this name
is retained as potential name. De¯ning what a reasonable
result for a digital library is crucial here. If a string is a
valid name, a number of papers are returned, otherwise the
output is either empty or with unlikely features. For
example when querying Citeseer with the term "Smith" more
than 10,000 papers are returned. This is the indication of
a potential anomaly: the probability that a person writes
more than 150 papers is quite low and the name can be
discarded. Equally, when looking for a non-name (e.g. the
words "Fortune Teller"), no papers are returned. We tend to
use quite restrictive criteria for keeping reliability high (e.g.
more than 5 papers and less than 50 returned by Citeseer);
redundancy of information allows to bootstrap learning
using just a limited amount of information, as already noted
by Brin [
All the occurrences of seed names are then annotated on the site's documents. Learning is performed initially only on documents where a reasonable quantity of known names are organized in HTML structures such as lists and tables. Such structures generally have an intrinsic semantic: lists generally contain elements of the same type (e.g. names of people), while the semantics in tables is generally related to the position either in rows or columns (e.g. all the elements of the ¯rst column are people, the second column represents addresses, etc.). When some elements (at least four or ¯ve in our case) are identi¯ed in a list or speci¯c portions of a table, we train a classi¯er able to relate a large part of these examples, for example using linguistic and/or formatting criteria (e.g. relevant names are always the ¯rst element in each row). If we succeed, we are able to reliably recognize other names in the structure. 3. Every department generally has one or more pages listing their sta® in some kind of lists. These are the lists that we are mainly looking for, but also tables assigning supervisors and students are useful, provided that students and teachers can be discriminated. Each time new examples are identi¯ed, the site is further annotated and more patterns can potentially be learnt. New names can be cross-checked on the resources used to identify the seed list: we have now more evidence that these names are real names. In our experiments this is enough to discover a large part of the sta® of an average CS website with very limited noise, even using a very strict strategy of multiple cross-evidence. We are currently using combinations of the following evidence to accept a learnt name: 1. the name was recognized as seed; 2. the name is included in an HTML structure where other known occurrences are found 3. there is an hyperlink internal to the site that wraps the whole name; 4. there is evidence from generic patterns (as derived by recognizing people on other sites) that this is a person.
3.0.4
To extract personal data (email address, telephone number, position, etc.) it is necessary to identify a dedicated web page (e.g. a personal web page). Again we combine information from Citeseer, HomepageSearch and Google to check if the person has a known page in the current web site. Otherwise we look for occurrences in the site in which the name is completely included in an hyperlink pointing internally to the site. It is then possible to extract personal data from the home page using a named entity recognizer (e.g. Annie) to easily identify them. In case some of the personal data are not found in the main page, subpages pointed by the home page are inspected. Only pages with an address under the same path are considered (e.g. www.aaa.edu/~adomine/index.html and www.aaa.edu/~adomine/contact.html refer to the same subdirectory. 3.0.5
Discovering what papers are written by what members of
the departmental sta® is a very di±cult task. It requires
3Classi¯ers are induced in our implementation by Amilcare,
http://nlp.shef.ac.uk/amilcare/, an adaptive IE system
developed at She±eld [
In order to bootstrap learning we query the digital libraries (Citeseer and UniTrier) using sta® names as keywords. The output for each name is hopefully a list of papers. Such lists will be incomplete because the digital libraries are largely incomplete. The titles in the list are then used to query a search engine to retrieve pages containing multiple paper citations. We focus on lists and tables where at least four papers are found. We use titles because they tend to be unique identi¯er. We are looking for seed examples, so we can discard titles which report too many hits (so to avoid titles which are very common strings such as "Lost"). As for discovering new names, the seed examples are annotated and page-speci¯c patterns are induced. We favour examples contained in HTML structures such as lists and tables for which we have multiple evidence. Please note however that the structure of the citation is often not very structured internally. For example <li>
Fabio Ciravegna, Alexiei Dingli, Daniela Petrelli and Yorick Wilks: <br> User-System Cooperation in Document Annotation based on Information Extraction <br> in Asuncion Gomez-Perez, V. Richard Benjamins (eds.): Knowledge Engineering and Knowledge Management (Ontologies and the Semantic Web), <br> Proceedings of the 13th International Conference on Knowledge Engineering and Knowledge Management (EKAW02), 1-4 October 2002 - Sigenza (Spain), Lecture Notes in Artificial Intelligence 2473, Springer Verlag <br> </li>
Simple wrappers relying on the HTML structure only
would be ine®ective, as there is no way to discriminate
for example - between authors and editors and title of paper
and title of collection using the HTML structure only. More
sophisticated wrapper induction systems are needed, as that
provided by Amilcare [
The architecture mentioned is fully implemented and we are currently experimenting extensively on a number of CS web sites. We have experimented on the sites of (1) the Computer Science Department of the University of She±eld (www.dcs.shef.ac.uk), (2) the Department of Engineering and Computer Science of the University of Southampton, UK, and (3) the Department of CS of the University of Aberdeen, UK. In the following we extensively report about the results on the site of the University of She±eld. Results on the other sites are qualitatively largely equivalent. 4.1
Experiments show that names of people can be found with a high reliability: in the case of the She±eld's department the system discovers 51 seed names of people belonging to the department as either academics, researchers or PhD students, 48 correct and 9 wrong. These names are used to the seed learning. Amilcare then discovers other 57 names, 48 correct, 6 wrong. This increases the overall recall from 37% to 74% with a very limited loss in precision (see Table 1). A qualitative evaluation of the errors is worth doing: the 9 false positives were: A. Schri±n, Eugenio Moggi, Peter Gray, Speech and Hearing, European Network, Department Of, Position Paper, The Network, To System. The ¯rst three are actually names of people; their recognition is due to the wrong identi¯cation of some web pages within the site as their home pages. They are all people for which Citeseer returns papers and they were all wrongly recognized at seeding time. The others (all recognized by the IE engine) are truly spurious hits for which a home page was also erroneously found. Their number is quite low in proportion to the total number of names. Recognizing these names as false hits is quite easy for a person, so if the results are to be used or checked by a person, they do not constitute a problem. One of them (Speech and Hearing) is actually the name of a group with its own home page. Concerning personal data, we just checked the identi¯cation of the correct home page; as people are accepted only if a home page is found, the quantitative results for people are also valid for home page identi¯cation. 4.2
As mentioned discovering papers is a very complex task. In our experiments a paper was considered correctly assigned to a person if it was a paper mentioned in the personal papers list of the author and the title is 100% correct. We did not use reseeding in the experiment, i.e. if a paper was coauthored by two researchers, the information returned for one person was not used to further annotate the publication pages for the second person. In this sense the redundancy of information was not fully exploited. For She±eld, the seed procedure discovers 488 papers, the learning procedure returns 752. We did not attempt discovery of further papers for people with less than 3 examples. Checking their correctness is very labor intensive, therefore we randomly checked the papers extracted for 8 of the sta® members for which the seed papers exceeded 6 examples; they are shown in Tables 2 and 3.
The use of IE increases signi¯cantly the overall recall rate which grows from 37 to 55 (+18%). Total recall is 47 for seeds and 66 for IE-based, precision 99.3 and 98.6 and Fmeasure 64.0 and 79.7 respectively (see Table 4). 5.
In this paper we have proposed a methodology to extract information from large repositories (e.g. large Web sites) with minimum user intervention. Information is initially extracted by starting from highly reliable/easy-to-mine sources such as databases and digital libraries and is then used to bootstrap more complex modules such as wrappers for extracting information from highly regular Web pages. Information extracted by the wrappers is then used to train more sophisticated IE engines. All the training corpora for the IE engines are produced automatically. Experimental results show that the methodology can produce high quality results.
The user intervention is limited to provide an initial URL and to add information missed by the di®erent modules when the computation is ¯nished. No preliminary manual annotation is required. The information added or delete by the user can then be reused for restarting learning and therefore getting more information (recall) and/or more precision. The type of user needed is a person able to understand the annotation task. No skills in IE are needed.
The natural application of such methodology is the Web, but large companies' repositories are also an option. In this paper we have focused on the use of the technology for mining web sites, an issue that can become very relevant for the Semantic Web, especially because annotation is provided largely without user intervention. It could potentially provide a partial solution to the outstanding problem of who is providing semantic annotation for the SW. It can potentially be used either by search engines associated to services/ontologies to automatically annotate/index/retrieve relevant documents or by speci¯c users to retrieve needed information on the °y by composing an architecture.
The idea of using the redundancy of information to
bootstrap IE learning is not new, having been already proposed
by Brin [
R1 R2 R3 R4 R5 R6 R7
R4 R5 R6 R7 R8 Total 33 46 12 77 32 36 84 320
seed discovery IE-based discovery 129 129 51 105 48 96 3 9 0 33 94 91 37 74 51 87 levels of complexity. In this respect our approach is - to our knowledge - unique. As noted by Brin, great care is needed in order to select only reliable information for annotation for learning. The integration of di®erent knowledge sources multiplies the available information, and therefore allows to seed learning only when multiple evidence is found. The experimental results described in this papers were obtained by ¯xing high precision (>90%) and testing the obtained recall. Other con¯gurations are possible and the results could radically change. Further experiments are needed for verifying the behavior of the system in di®erent con¯gurations.
One relevant question for the e®ective usability of the methodology in real applications concerns the required level of accuracy (as a balance of precision and recall) the system has to provide. As Web applications are concerned, it is well known that high accuracy is not always required. Search engines are used every day by millions of people, even if their accuracy is far from ideal: further navigation is often required to ¯nd satisfying results, large portions of the Web are not indexed (the so called dark and invisible Webs), etc. Services like Citeseer, although incomplete, are a very successful. What really seems to matter is size: the ability to both retrieve information dispersed on the Web and create a critical mass or relatively reliable information. In this respect the proposed methodology is satisfying. Experiments show that in the case of paper discovery, it is able to discover a large part of the information that the digital libraries used to seed did not have (+50%). Precision is topping 90%, so the information provided is very reliable. 5.0.1
From the IE point of view there are a number of challenges in learning from automatic annotation, instead of using human annotation. On the one hand not all the annotation is reliable: the use of multiple strategies and combined evidence reduces the problem, but still there is a strong need for methodologies robust with respect to noise. On the other hand, many IE systems are able to learn from completely annotated documents only, so that all the annotated strings are considered positive examples and the rest of the text is used as a set of counterexamples. In our cycle of seed and learn, we generally produce partially annotated documents. This means that the system is presented with positive examples, but the rest of the texts can never be considered as a set of negative examples, because unannotated portions of text can contain instances that the system has to discover, not counterexamples. This is a challenge for the learner. At the moment we present the learner with just the annotated portion of the text plus a windows of words of context, not with the whole document. This is enough to have the system learning correctly: the unannotated examples that become negative examples entering the training corpus is generally low enough to avoid problems. In the future we will have to focus on using machine learning methodologies that are able to learn from scattered annotation. 5.0.2
The proposed methodology is based on using the
redundancy of information. Information is extracted from
di®erent sources (databases, digital libraries, documents, etc.),
therefore the classic problems of integrating information arise.
Information can be represented in di®erent ways in di®erent
sources from both a syntactic and a semantic point of view.
The syntactic variation is coped with in the de¯nition
architecture de¯nition step: when two modules are connected, a
canonical form of the information is de¯ned, e.g. the
classic problem of recognising ¯lm titles as "The big chill" and
"Big chill, the" can be addressed. More complex tasks are
to be addressed, though. For example, a person name can
be cited in di®erent ways: N. Weaver, Nick Weaver and
Nicholas Weaver are potential variation of the same name.
But do they identify the same person as well? When large
quantity of information is available (e.g. authors names in
Citeseer) this becomes an important issue [
This work was carried out within the AKT project (http://www.aktors.org), sponsored by the UK Engineering and Physical Sciences Research Council (grant GR/N15764 /01). AKT involves the Universities of Aberdeen, Edinburgh, She±eld, Southampton and the Open University. Its objectives are to develop advanced technologies for knowledge management. Thanks to David Guthrie for the contribution to the discussion.