=Paper=
{{Paper
|id=Vol-101/paper-1
|storemode=property
|title=Automatic Semantic Annotation Using Unsupervised Information Extraction and Integration
|pdfUrl=https://ceur-ws.org/Vol-101/Alexiei_Dingli-et-al.pdf
|volume=Vol-101
}}
==Automatic Semantic Annotation Using Unsupervised Information Extraction and Integration==
https://ceur-ws.org/Vol-101/Alexiei_Dingli-et-al.pdf
Automatic Semantic Annotation using Unsupervised
Information Extraction and Integration
Alexiei Dingli, Fabio Ciravegna, and Yorick Wilks
ABSTRACT irrelevant for some use(r)s: a page in a pet shop web site can
In this paper we propose a methodology to learn to auto- be annotated with shop-related annotations, but some users
matically annotate domain-specific information from large would rather prefer to find annotations related to animals.
repositories (e.g. Web sites) with minimum user interven- Different annotation can be imposed on a document us-
tion. The methodology is based on a combination of in- ing different ontologies. An ontology is required because it
formation extraction, information integration and machine describes concepts and relationships that occur in a very
learning techniques. Learning is seeded by extracting infor- restricted view of the real world, basically it describes the
mation from structured sources (e.g. databases and digital domain in which we are working. Most of the annotation is
libraries). Retrieved information is then used to partially likely to be associated in the future by Web actors other than
annotate documents. These annotated documents are used the page’s owner, exactly like nowadays’ search engines pro-
to bootstrap learning for simple Information Extraction (IE) duce indexes without modifying the page code. Producing
methodologies, which in turn will produce more annotations methodologies for automatic annotation of pages with no or
used to annotate more documents. It will be used to train minimal user intervention becomes therefore important: the
more complex IE engines and the cycle will keep on repeat- initial annotation associated to the document loses its im-
ing itself until the required information is obtained. The portance because at any time it is possible to automatically
user intervention is limited to providing an initial URL and (re)annotate the document and to store the annotation in a
to correct information if it is the case when the computation separate database or ontology. In the future Semantic Web,
is finished. The revised annotation can then be reused to automatic annotation systems might become as important
provide further training and therefore getting more informa- as indexing systems are nowadays for search engines.
tion and/or more precision. Automatic annotation methodologies have been developed
in the past at the intersection of research areas such as Infor-
mation Extraction from text (IE)[19], Information Integra-
1. INTRODUCTION tion (II)[12], wrapper induction [13] and machine learning
The Semantic Web (SW) needs semantically-based docu- [17]. Porting to new application domains in fairly uncon-
ment annotation1 to both enable better document retrieval strained areas such as the Web is out of reach for the current
and empower semantically-aware agents. Most of the cur- technologies. The association to domain-specific ontologies
rent technology is based on human centered annotation, very limits the domain and makes the application feasible. For
often completely manual [11]. Manual annotation is difficult, example IE is currently used to reduce the burden in some
time consuming and expensive [3]. Convincing millions of SW annotation tools[20] [10] [3] and used to crawl the Web
users to annotate documents for the Semantic Web is diffi- for harvesting domain specific information [14] [17]. Most
cult and requires a world-wide action of uncertain outcome. of this technology is based on supervised learning, i.e. they
In this framework, annotation is meant mainly to be stati- require user-defined annotated corpora. Producing such cor-
cally associated to (and saved within) the documents. Static pora can be difficult and time consuming, even when using
annotation associated to a document can: (1) be incomplete adaptive IE as support to annotation. If we take a look
or incorrect when the creator is not skilled enough; (2) be- at II, we find that there are many projects focusing on the
come obsolete, i.e. not be aligned with pages updates; (3) be web[9] [15] [12] but they all fall short when it comes to pro-
1 cess sites with very irregular formatting or containing free
Semantic annotation is the process of inserting tags in the
document, whose purpose is to assign semantics to the text text. If the documents are many and very different, and the
between the opening and closing tags. annotation to be performed very detailed, the annotation
process may require substantial work. In case of annotat-
ing large portions of the Web (e.g. groups of whole web
sites), the task could become unfeasible. Therefore, there is
the need of training the system without manually producing
Permission to make digital or hard copies of all or part of this work for the annotated material.
personal or classroom use is granted without fee provided that copies are In this paper we propose a methodology to learn how
not made or distributed for profit or commercial advantage and that copies to annotate semantically-consistent portions of the Web ex-
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
tracting and integrating information from different sources.
permission and/or a fee. All the annotation is produced automatically with no user
Copyright 2000 ACM 0-89791-88-6/97/05 ..$5.00
intervention apart some corrections the users might want to ble to extract information with different reliability. Systems
perform. The methodology has been fully implemented in such as databases generally contain structured data and can
Armadillo2 , a system for unsupervised information extrac- be queried using an API. In case the API is not available
tion and integration from large collections of documents. (e.g. the database has a web front end and the output is tex-
The natural application of such methodology is the Web, tual), wrappers can be induced to extract such information.
but large companies’ information systems are also an op- Wrapper Induction methodologies are able to model rigidly
tion. In this paper we will focus on the Web, and in partic- structured Web pages such as those produced by databases
ular in mining web sites of Computer Science Departments. [13] [18]. When the information is contained in textual doc-
All the process is based on integrating information from dif- uments, extracting information requires more sophisticated
ferent sources in order to provide some seed annotations. methodologies. Wrapper induction systems have been ex-
This will then bootstrap learning which in turn will provide tended to cope with less rigidly structured pages [8], free
more annotations and so on. In synthesis we start with a texts and even a mixture of them [4]. There is an obvious
simple methodology which requires limited annotation, and increasing degree of complexity in the extraction task men-
move on to produce further annotation to train more com- tioned above. The more the task is difficult, the less reliable
plex modules. generally the extracted information is. For example wrap-
In the next section we will present the generic architec- per induction systems generally reach 100% on rigidly struc-
ture that we have used to build the application. Then we tured documents, while IE systems reach some 70% on free
will describe the CS Department task and how the informa- texts. Also, the more the complexity increases, the more the
tion from different sources is integrated in order to learn to amount of data needed for training grows: wrappers can be
annotate the desired information. Experimental results are trained with a handful of examples whereas full IE systems
illustrated and finally we will discuss some future challenges may require millions of words [16]. This is just an example
that our experience highlights. of the idea. The more the task becomes complex, the more
information is needed for training, the more reliable input
data becomes difficult to identify.
2. ARMADILLO: To make system scalable we have implemented an archi-
A GENERIC ARCHITECTURE tecture based on Web Services where each task is divided in
FOR WEBSITE MINING subtasks. Each subtask is performed by a server which in
turn will use other servers for implementing parts of the sub-
Since there can be many potential applications of the pro- task. Each server exposes a declaration of input and output,
posed technology we decided to make the Architecture as plus a set of working parameters. Servers are reusable in dif-
generic as possible. In order to do so, the Architecture had ferent contexts and applications. For example one server in
to be portable and scalable. Portability was achieved by the CS department task will return all papers written by
making use of a simple methodology. The system starts from a person by accessing Citeseer. Another one will do the
a set of generic strategies defining where to search informa- same on another digital library . The named entity recog-
tion and what to look for. When data is harvested using niser server (whose role is to decide if a string is a name)
those strategies, it is passed to an oracle in order to verify will invoke these servers and integrate the evidence returned
whether it is valid information or not. By oracle we mean a and decide if such evidence is enough to conclude that the
human or a resource (such as an IE engine, a database etc.) candidate string represents a person.
that can identify which items on a web page are instances of Facilities for defining wrappers are provided in our archi-
concepts found in a preloaded ontology. Once we have some tecture by Amilcare (nlp.shef.ac.uk/amilcare/), an adaptive
information, the system applies further strategies which can IE system based on a wrapper induction methodology able
be used with that kind of information and the results are to cope with a whole range of documents from rigidly struc-
given to other more specialised oracles. Basically the sys- tured documents to free texts [6]. Amilcare can be trained to
tem keeps on looping until there is no more information to work on rigid documents (e.g. Citeseer or Google output) by
discover or the user decides to interrupt the cycle. It is clear providing a handful of manually annotated examples, while
that we don’t make any assumptions on any domain. The it needs some hundreds of examples for more sophisticated
strategies range from simple/generic ones (like looking for cases [5]. All the servers are defined in a resource pool and
capitalised words) to more complex ones (like setting up and can be inserted in a user-defined architecture to perform
using IE tools) and they can be easily created for any do- some specific tasks. New servers can be defined and added
main. This technique works because a key feature of the web to the pool by wrapping them in a standard format. In the
is the Redundancy of information. Redundancy is given by CS website task, wrappers are defined for all the resources
the presence of multiple citations of the same information in described in Section 3. The CS application works in the
different contexts and in different superficial formats. This following way: a user submits a URL. The system returns
factor is currently used for improving question answering a database populated with people’s names, personal details,
systems [7]. When known information is present in different papers, projects, etc. The defined architecture works as a
sources, it is possible to use its multiple occurrences to boot- ”Glass Box”. All the steps performed by the system are
strap recognizers that, when generalized, will retrieve other shown to the user together with their input and output.
pieces of information, producing in turn more (generic) rec- The user can check the intermediate results and manually
ognizers [2]. Information can be present in different formats modify their output, or change their strategy (if possible,
on the Web: in documents, in repositories (e.g. databases or such as in the case of modules who integrate information).
digital libraries), via agents able to integrate different infor- For example if a person name is missed by the system, it can
mation sources, etc. From them or their output, it is possi- be manually added by the user. The modules that receive as
2 input the output of that name finder will then be re-run and
http://www.aktors.org/technologies/Armadillo/
further information will hopefully be retrieved. In this way – Input: the potential name;
the user is able both to check the results of each step and – Output: a URL for home page (if any);
to improve the results of the system by manually providing
some contributions (additions, corrections, deletion). • Annie
(www.gate.ac.uk):
3. THE COMPUTER SCIENCE – Input: the potential name and the text surround-
DEPARTMENT TASK ing it;
The application used to describe the methodology is min- – Output: True/False;
ing websites of Computer Science Departments. The goal
is to discover who works in a specific department (name, • Google
position, home page, email address, telephone number) and (www.google.co.uk)
to extract for each person some personal data and a list of
published papers larger than the one provided by services – Input: the potential name and the URL of the
such as Citeseer. The structure and layout of the web pages site in order to restrict search;
is different for every department but we can expect to find – Output: Relevant Pages that are hopefully home
some common elements in Computer Science Departments pages;
web pages such as a people’s page containing the list of peo-
ple working in the department and many others. The information returned by the digital libraries (Citeseer
and Unitrier) is used to confirm or deny the name identity
3.0.1 Finding People Names of the string. If they return reasonable results for a spe-
The goal of this task is to discover the names of the peo- cific name (i.e. not too few and not too many), this name
ple who work in the specific department. This task is more is retained as potential name. Defining what a reasonable
complex than a generic Named Entity Recognition because result for a digital library is crucial here. If a string is a
many irrelevant people’s names are cited in a site, e.g. names valid name, a number of papers are returned, otherwise the
of undergraduate students, clerics, secretaries, etc, as well as output is either empty or with unlikely features. For ex-
names of researchers from other sites that e.g. participate in ample when querying Citeseer with the term ”Smith” more
common projects or have co-authored papers with members than 10,000 papers are returned. This is the indication of
of staff. Organizing the extraction around a generic Named a potential anomaly: the probability that a person writes
Entity Recognizer (NER) such as Annie (www.gate.ac.uk) more than 150 papers is quite low and the name can be
is the most natural option. This does not finish the job, discarded. Equally, when looking for a non-name (e.g. the
though, because a NER recognizes ALL the people’s names words ”Fortune Teller”), no papers are returned. We tend to
in the site, without discriminating between relevant and ir- use quite restrictive criteria for keeping reliability high (e.g.
relevant. Moreover classic NER tend to be quite slow if more than 5 papers and less than 50 returned by Citeseer);
launched on large sites (e.g. the 1,600 pages of the CS de- redundancy of information allows to bootstrap learning us-
partment at the University of Southampton) and can be ing just a limited amount of information, as already noted
quite imprecise on Web pages, as they are generally defined by Brin [2]. The results of the digital libraries are integrated
for newspaper-like articles. A two-step strategy is used here with those of the classic Named Entity Recognizer run on a
instead: initially a short list of highly reliable seed names window of words around the candidate (so to avoid the prob-
are found. Then these seeds are used to bootstrap learning lem of slow processing). At this point a number of names of
for finding further names. people are available. They are in principle of three types:
3.0.2 Finding Seed Names 1. correct (they are people working for this department);
To find seed names, a number of weak strategies are com- 2. wrong (they are not people: they are false positives);
bined that integrate information from different sources. First
all the web site is crawled looking for strings that are poten- 3. people who do not work at this site, but that are cited
tial names of people (e.g. using a gazetteer of first names because, for example, they have coauthored papers
and a regular expression such as + (capitalized with some of the researchers of the department.
word)+ .). Then the following web services are queried:
For this reason, Citeseer, Google and HomepageSearch are
• Citeseer used to look for a personal web page in the site. If such a
(www.citeseer.com): page is not found, the names are discarded. From the re-
sults, personal web pages are recognized with simple heuris-
– Input: the potential name;
tics such as looking for the name in the title or in ”< H1 >”
– Output: a list of papers and a URL for home page tags. The process mentioned above is meant to determine a
(if any); small, highly reliable list of seed names to enable learning.
Each of the strategies is, per se, weak, as they all report high
• The CS bibliography at Unitrier
recall, low precision. Their combination is good enough to
(http://www.informatik.uni-trier.de/ ley/db/):
produce data with high accuracy.
– Input: the potential name:
3.0.3 Learning Further Names
– Output: a list of papers (if any);
All the occurrences of seed names are then annotated on
• HomePageSearch the site’s documents. Learning is performed initially only
(http://hpsearch.uni-trier.de/): on documents where a reasonable quantity of known names
are organized in HTML structures such as lists and tables. to identify the paper title and the authors, then relate the
Such structures generally have an intrinsic semantic: lists authors to the people identified in the previous step. Au-
generally contain elements of the same type (e.g. names of thors are names in particular positions and in particular
people), while the semantics in tables is generally related to contexts: they must not be confused with editors of collec-
the position either in rows or columns (e.g. all the elements tions in which the paper can be published, nor they must
of the first column are people, the second column represents be confused with other names mentioned in the surrounding
addresses, etc.). When some elements (at least four or five text. A title is generally a random sequence of words (e.g.
in our case) are identified in a list or specific portions of a the title of [7]) and cannot be characterized in any way (i.e.
table, we train a classifier able to relate a large part of these we cannot write generic patterns for identifying candidate
examples, for example using linguistic and/or formatting strings as we did for people). Moreover paper titles must not
criteria (e.g. relevant names are always the first element in be confused with titles of collections in which they are pub-
each row). If we succeed, we are able to reliably recognize lished. Nearly each department and each member of staff
other names in the structure. 3 . Every department generally in CS departments provide a list of publications. Moreover
has one or more pages listing their staff in some kind of papers are co-authored, so it is very possible that each paper
lists. These are the lists that we are mainly looking for, but is cited more than one time in a specific site. In rare cases
also tables assigning supervisors and students are useful, personal lists of papers are produced using a departmental
provided that students and teachers can be discriminated. database (i.e. all the publication pages are formatted in the
Each time new examples are identified, the site is further same way), but in most cases each person writes the list
annotated and more patterns can potentially be learnt. New using a personal format; very often the style is quite irregu-
names can be cross-checked on the resources used to identify lar as the list is compiled manually in different moments of
the seed list: we have now more evidence that these names times. This is a typical case in which the classic methodol-
are real names. In our experiments this is enough to discover ogy of manually annotating some examples for each page for
a large part of the staff of an average CS website with very each member of staff is unfeasible, due to the large number
limited noise, even using a very strict strategy of multiple of different pages. Also irregularities in style produce noisy
cross-evidence. We are currently using combinations of the data and classic wrappers are not able to cope with noise.
following evidence to accept a learnt name: A generic methodology is needed that does not require any
manual annotation.
1. the name was recognized as seed; In order to bootstrap learning we query the digital li-
braries (Citeseer and UniTrier) using staff names as key-
2. the name is included in an HTML structure where words. The output for each name is hopefully a list of pa-
other known occurrences are found pers. Such lists will be incomplete because the digital li-
braries are largely incomplete. The titles in the list are then
3. there is an hyperlink internal to the site that wraps used to query a search engine to retrieve pages containing
the whole name; multiple paper citations. We focus on lists and tables where
at least four papers are found. We use titles because they
4. there is evidence from generic patterns (as derived by
tend to be unique identifier. We are looking for seed ex-
recognizing people on other sites) that this is a person. amples, so we can discard titles which report too many hits
The latter strategy was inspired by [17]. (so to avoid titles which are very common strings such as
”Lost”). As for discovering new names, the seed examples
3.0.4 Extracting Personal Data are annotated and page-specific patterns are induced. We
favour examples contained in HTML structures such as lists
To extract personal data (email address, telephone num- and tables for which we have multiple evidence. Please note
ber, position, etc.) it is necessary to identify a dedicated however that the structure of the citation is often not very
web page (e.g. a personal web page). Again we combine structured internally. For example
information from Citeseer, HomepageSearch and Google to
check if the person has a known page in the current web
site. Otherwise we look for occurrences in the site in which Fabio Ciravegna,
the name is completely included in an hyperlink pointing Alexiei Dingli,
internally to the site. It is then possible to extract per- Daniela Petrelli and
sonal data from the home page using a named entity rec- Yorick Wilks:
ognizer (e.g. Annie) to easily identify them. In case some
of the personal data are not found in the main page, sub- User-System Cooperation in Document Annotation
pages pointed by the home page are inspected. Only pages based on Information Extraction
with an address under the same path are considered (e.g.
www.aaa.edu/ãdomine/index.html and in Asuncion Gomez-Perez, V. Richard Benjamins
www.aaa.edu/ãdomine/contact.html refer to the same (eds.): Knowledge Engineering and Knowledge
subdirectory. Management (Ontologies and the Semantic Web),
3.0.5 Discovering Papers Citations Proceedings of the 13th International
Discovering what papers are written by what members of Conference on Knowledge Engineering and
the departmental staff is a very difficult task. It requires Knowledge Management (EKAW02), 1-4 October 2002
- Sigenza (Spain), Lecture Notes in Artificial
3
Classifiers are induced in our implementation by Amilcare, Intelligence 2473, Springer Verlag
http://nlp.shef.ac.uk/amilcare/, an adaptive IE system devel-
oped at Sheffield [6]
Simple wrappers relying on the HTML structure only did not use reseeding in the experiment, i.e. if a paper was
would be ineffective, as there is no way to discriminate - coauthored by two researchers, the information returned for
for example - between authors and editors and title of paper one person was not used to further annotate the publication
and title of collection using the HTML structure only. More pages for the second person. In this sense the redundancy
sophisticated wrapper induction systems are needed, as that of information was not fully exploited. For Sheffield, the
provided by Amilcare [6], which exploits both XML struc- seed procedure discovers 488 papers, the learning procedure
tures and (para-)linguistic information [5]. Using a cycle returns 752. We did not attempt discovery of further pa-
of annotation/learning/annotation we are able to discover a pers for people with less than 3 examples. Checking their
large number of new papers. The cycle finishes when there is correctness is very labor intensive, therefore we randomly
no more information to discover or the user decides to inter- checked the papers extracted for 8 of the staff members for
rupt the cycle. Note that every time co-authorship among which the seed papers exceeded 6 examples; they are shown
people is discovered in analysing the publication page of one in Tables 2 and 3.
specific authors, the paper is retained for annotation when The use of IE increases significantly the overall recall rate
the other names are considered (i.e. the redundancy is ex- which grows from 37 to 55 (+18%). Total recall is 47 for
ploited again). seeds and 66 for IE-based, precision 99.3 and 98.6 and F-
measure 64.0 and 79.7 respectively (see Table 4).
4. EVALUATION
The architecture mentioned is fully implemented and we 5. CONCLUSION AND FUTURE WORK
are currently experimenting extensively on a number of CS In this paper we have proposed a methodology to extract
web sites. We have experimented on the sites of (1) the information from large repositories (e.g. large Web sites)
Computer Science Department of the University of Sheffield with minimum user intervention. Information is initially ex-
(www.dcs.shef.ac.uk), (2) the Department of Engineering tracted by starting from highly reliable/easy-to-mine sources
and Computer Science of the University of Southampton, such as databases and digital libraries and is then used to
UK, and (3) the Department of CS of the University of Ab- bootstrap more complex modules such as wrappers for ex-
erdeen, UK. In the following we extensively report about the tracting information from highly regular Web pages. In-
results on the site of the University of Sheffield. Results on formation extracted by the wrappers is then used to train
the other sites are qualitatively largely equivalent. more sophisticated IE engines. All the training corpora for
the IE engines are produced automatically. Experimental
4.1 Finding People’s Names results show that the methodology can produce high quality
Experiments show that names of people can be found with results.
a high reliability: in the case of the Sheffield’s department The user intervention is limited to provide an initial URL
the system discovers 51 seed names of people belonging to and to add information missed by the different modules
the department as either academics, researchers or PhD stu- when the computation is finished. No preliminary manual
dents, 48 correct and 9 wrong. These names are used to the annotation is required. The information added or delete by
seed learning. Amilcare then discovers other 57 names, 48 the user can then be reused for restarting learning and there-
correct, 6 wrong. This increases the overall recall from 37% fore getting more information (recall) and/or more precision.
to 74% with a very limited loss in precision (see Table 1). The type of user needed is a person able to understand the
A qualitative evaluation of the errors is worth doing: the annotation task. No skills in IE are needed.
9 false positives were: A. Schriffin, Eugenio Moggi, Peter The natural application of such methodology is the Web,
Gray, Speech and Hearing, European Network, Department but large companies’ repositories are also an option. In this
Of, Position Paper, The Network, To System. The first three paper we have focused on the use of the technology for min-
are actually names of people; their recognition is due to the ing web sites, an issue that can become very relevant for the
wrong identification of some web pages within the site as Semantic Web, especially because annotation is provided
their home pages. They are all people for which Citeseer re- largely without user intervention. It could potentially pro-
turns papers and they were all wrongly recognized at seeding vide a partial solution to the outstanding problem of who
time. The others (all recognized by the IE engine) are truly is providing semantic annotation for the SW. It can poten-
spurious hits for which a home page was also erroneously tially be used either by search engines associated to ser-
found. Their number is quite low in proportion to the to- vices/ontologies to automatically annotate/index/retrieve rel-
tal number of names. Recognizing these names as false hits evant documents or by specific users to retrieve needed in-
is quite easy for a person, so if the results are to be used formation on the fly by composing an architecture.
or checked by a person, they do not constitute a problem. The idea of using the redundancy of information to boot-
One of them (Speech and Hearing) is actually the name of strap IE learning is not new, having been already proposed
a group with its own home page. Concerning personal data, by Brin [2] and Mitchell [17]. The difference with our ap-
we just checked the identification of the correct home page; proach is the way in which learning is bootstrapped. Brin
as people are accepted only if a home page is found, the uses user-defined examples, while Mitchell uses generic pat-
quantitative results for people are also valid for home page terns that work independently from the place at hand (e.g.
identification. the site or the page). We use both the above4 , but in addi-
tion, we exploit the redundancy of information and integrate
4.2 Paper Discovery information extracted from different sources with different
As mentioned discovering papers is a very complex task.
In our experiments a paper was considered correctly as- 4
Generic patterns are used in the named entity recognizer.
signed to a person if it was a paper mentioned in the personal User-defined examples are used to discover project names,
papers list of the author and the title is 100% correct. We a task not described in the paper.
Possible Actual Correct Wrong Missing Precision Recall F-Measure
seed discovery 129 51 48 3 0 94 37 51
IE-based discovery 129 105 96 9 33 91 74 87
Table 1: Results in Discovering People and Associated Home Page. The first line refers to accuracy in
discovering names using the procedure for seed names (Citeseer+Google, etc.), the second one to the discovery
using adaptive IE.
Possible Actual Correct Wrong Missing Precision Recall F-Measure
R1 33 14 14 0 19 100 42 59
R2 46 23 22 1 23 95 47 63
R3 12 7 7 0 5 100 58 73
R4 77 31 31 0 46 100 40 57
R5 32 15 15 0 17 100 46 63
R6 36 15 15 0 21 100 41 58
R7 84 47 47 0 37 100 56 71
Total 320 152 151 1 168 99 47 64
Table 2: Seed Paper Discovery Accuracy; each line represents the papers discovered for a person. Possible
represents the number of papers present in the personal publication list page, Actual the number of papers
returned by the systems. Actual results are divided in Correct, Wrong and Missing.
Possible Actual Correct Wrong Missing Precision Recall F-Measure
R1 33 20 20 0 13 100 60 75
R2 46 36 36 0 10 100 78 87
R3 12 11 11 0 1 100 91 95
R4 77 38 38 0 39 100 49 66
R5 32 25 25 0 7 100 78 87
R6 36 16 15 1 20 93 41 57
R7 100 22 19 3 78 86 19 31
R8 84 71 69 2 13 97 82 89
Total 420 239 233 6 181 97 55 70
Table 3: IE-based Paper Discovery Accuracy obtained by Amilcare using the seeds in table 2 to bootstrap
learning.
Possible Actual Correct Wrong Missing Precision Recall F-Measure
seed 320 152 151 1 168 99.3 47.2 64.0
IE-based 320 217 214 3 103 98.6 66.9 79.7
Table 4: Paper Discovery: Grand total
levels of complexity. In this respect our approach is - to our The syntactic variation is coped with in the definition archi-
knowledge - unique. As noted by Brin, great care is needed tecture definition step: when two modules are connected, a
in order to select only reliable information for annotation canonical form of the information is defined, e.g. the clas-
for learning. The integration of different knowledge sources sic problem of recognising film titles as ”The big chill” and
multiplies the available information, and therefore allows to ”Big chill, the” can be addressed. More complex tasks are
seed learning only when multiple evidence is found. The ex- to be addressed, though. For example, a person name can
perimental results described in this papers were obtained by be cited in different ways: N. Weaver, Nick Weaver and
fixing high precision (>90%) and testing the obtained recall. Nicholas Weaver are potential variation of the same name.
Other configurations are possible and the results could rad- But do they identify the same person as well? When large
ically change. Further experiments are needed for verifying quantity of information is available (e.g. authors names in
the behavior of the system in different configurations. Citeseer) this becomes an important issue [1]. This problem
One relevant question for the effective usability of the intersects with that of intra- and inter-document corefer-
methodology in real applications concerns the required level ence resolution well known in Natural Language Processing.
of accuracy (as a balance of precision and recall) the system We are currently focusing on mining websites, because this
has to provide. As Web applications are concerned, it is well allows to apply some heuristics that very often solve these
known that high accuracy is not always required. Search en- problems in a satisfying way. For example the probability
gines are used every day by millions of people, even if their that N. Weaver, Nick Weaver and Nicholas Weaver are not
accuracy is far from ideal: further navigation is often re- the same person in a specific CS website is very low and
quired to find satisfying results, large portions of the Web therefore it is possible to hypothesize coreference. Different
are not indexed (the so called dark and invisible Webs), etc. is the case of ambiguity in the external resources (e.g. in the
Services like Citeseer, although incomplete, are a very suc- digital libraries). Here the problem is more pervasive. When
cessful. What really seems to matter is size: the ability to querying with very common names (e.g. ”John Smith”) the
both retrieve information dispersed on the Web and create results is quite disappointing, as papers by different people
a critical mass or relatively reliable information. In this re- are mixed. This is not a problem in our approach because
spect the proposed methodology is satisfying. Experiments the information returned is used to annotate the site. Paper
show that in the case of paper discovery, it is able to discover from people from other departments or universities will not
a large part of the information that the digital libraries used introduce any annotations and therefore will not cause any
to seed did not have (+50%). Precision is topping 90%, so problems. The same applies in case multiple home pages are
the information provided is very reliable. returned: if they do not have an address local to the current
site, the page is not used. In the generic case, though, this
5.0.1 Challenges for IE is a problem. We are currently using this strategy to rec-
From the IE point of view there are a number of challenges ognize named entities from Reuters news and to find more
in learning from automatic annotation, instead of using hu- information about a specific name. In this case we have to
man annotation. On the one hand not all the annotation understand if the Ken Russell cited in a specific news is the
is reliable: the use of multiple strategies and combined evi- famous regissuer or an MTI researcher. We are currently
dence reduces the problem, but still there is a strong need for experimenting with a strategy that integrates also evidence
methodologies robust with respect to noise. On the other from lexical chains extracted from generic ontologies. The
hand, many IE systems are able to learn from completely idea is that an MTI researcher and a regisseur should pro-
annotated documents only, so that all the annotated strings duce different lexical chains (one concerning computers, the
are considered positive examples and the rest of the text is other concerning films).
used as a set of counterexamples. In our cycle of seed and
learn, we generally produce partially annotated documents. Acknowledgements
This means that the system is presented with positive ex- This work was carried out within the AKT project
amples, but the rest of the texts can never be considered as (http://www.aktors.org), sponsored by the UK Engineering
a set of negative examples, because unannotated portions of and Physical Sciences Research Council (grant GR/N15764
text can contain instances that the system has to discover, /01). AKT involves the Universities of Aberdeen, Edin-
not counterexamples. This is a challenge for the learner. At burgh, Sheffield, Southampton and the Open University. Its
the moment we present the learner with just the annotated objectives are to develop advanced technologies for knowl-
portion of the text plus a windows of words of context, not edge management. Thanks to David Guthrie for the contri-
with the whole document. This is enough to have the system bution to the discussion.
learning correctly: the unannotated examples that become
negative examples entering the training corpus is generally 6. REFERENCES
low enough to avoid problems. In the future we will have [1] H. Alani, S. Dasmahapatra, N. Gibbins, H. Glaser,
to focus on using machine learning methodologies that are S. Harris, Y. Kalfoglou, K. O’Hara, and N. Shadbolt.
able to learn from scattered annotation. Managing reference: Ensuring referential integrity of
ontologies for the semantic web. In Proceedings of the
5.0.2 Integrating Information from Different Sources 13th International Conference on Knowledge
The proposed methodology is based on using the redun- Engineering and Knowledge Management, EKAW02.
dancy of information. Information is extracted from differ- Springer Verlag, 2002.
ent sources (databases, digital libraries, documents, etc.), [2] Sergey Brin. Extracting patterns and relations from
therefore the classic problems of integrating information arise. the world wide web. In WebDB Workshop at 6th
Information can be represented in different ways in different International Conference on Extending Database
sources from both a syntactic and a semantic point of view. Technology, EDBT’98, 1998.
[3] F. Ciravegna, Alexiei Dingli, Daniela Petrelli, and [15] A. Keller M. Genesereth and O. Duschka. Infomaster:
Yorick Wilks. User-system cooperation in document An information integration system. In Proceedings of
annotation based on information extraction. In the ACM SIGMOD International Conference on
Proceedings of the 13th International Conference on Management of Data, Tucson, AZ, 1997, 1997.
Knowledge Engineering and Knowledge Management, [16] S. Miller, M. Crystal, H. Fox, L. Ramshaw,
EKAW02. Springer Verlag, 2002. R. Schwartz, R. Stone, and R. Weischedel. Bbn:
[4] Fabio Ciravegna. Adaptive information extraction Description of the sift system as used for MUC7. In
from text by rule induction and generalisation. In Proceedings of the 7th Message Understanding
Proceedings of 17th International Joint Conference on Conference, 1998.
Artificial Intelligence (IJCAI), 2001. Seattle. www.itl.nist.gov/iaui/894.02/relatedp rojects/muc/.
[5] Fabio Ciravegna. (LP)2 , an adaptive algorithm for [17] Tom Mitchell. Extracting targeted data from the web.
information extraction from web-related texts. In In Proceedings of the seventh ACM SIGKDD
Proceedings of the IJCAI-2001 Workshop on Adaptive international conference on Knowledge discovery and
Text Extraction and Mining held in conjunction with data mining, San Francisco, California, 2001.
the 17th International Joint Conference on Artificial [18] I. Muslea, S. Minton, and C. Knoblock. Wrapper
Intelligence, 2001. Seattle, induction for semistructured web-based information
http://www.smi.ucd.ie/ATEM2001/. sources. In Proceedings of the Conference on
[6] Fabio Ciravegna. Designing adaptive information Automated Learning and Discovery (CONALD),
extraction for the semantic web in amilcare. In 1998., 1998.
S. Handschuh and S. Staab, editors, Annotation for [19] Maria Teresa Pazienza, editor. Information Extraction:
the Semantic Web, Frontiers in Artificial Intelligence A multidisciplinary approach to an emerging
and Applications. IOS Press, Amsterdam, 2003. information technology. Springer Verlag, 1999.
[7] Susan Dumais, Michele Banko, Eric Brill, Jimmy Lin, [20] M. Vargas-Vera, Enrico Motta, J. Domingue,
and Andrew Ng. Web question answering: Is more M. Lanzoni, A. Stutt, and F. Ciravegna. MnM:
always better? In Proceedings of the 25th Annual Ontology driven semi-automatic or automatic support
International ACM SIGIR Conference on Research for semantic markup. In Proceedings of the 13th
and Development in Information Retrieval (SIGIR International Conference on Knowledge Engineering
2002), Tampere, Finland, 2002. and Knowledge Management, EKAW02. Springer
[8] D. Freitag and N. Kushmerick. Boosted wrapper Verlag, 2002.
induction. In R. Basili, F. Ciravegna, and
R. Gaizauskas, editors, ECAI2000 Workshop on
Machine Learning for Information Extraction, 2000.
www.dcs.shef.ac.uk/ fabio/ecai-workshop.html.
[9] A. Masci P. Merialdo G. Mecca, P. Atzeni and
G. Sindoni. From databases to web-bases: The
araneus experience. In Technical Report 34-1998.
Dipartimento di Informatica e Automazione,
Universita’ di Roma Tre, 1998.
[10] S. Handschuh, S. Staab, and F. Ciravegna. S-CREAM
- Semi-automatic CREAtion of Metadata. In
Proceedings of the 13th International Conference on
Knowledge Engineering and Knowledge Management,
EKAW02. Springer Verlag, 2002.
[11] S. Handschuh, S. Staab, and A. Maedche. CREAM —
Creating relational metadata with a component-based,
ontology driven framework. In In Proceedings of
K-Cap 2001, Victoria, BC, Canada, October 2001.
[12] C. Knoblock, S. Minton, J.L. Ambite, N. Ashish,
J. Margulis, J. Modi, I. Muslea, A. Philpot, and
S. Tejada. The ariadne approach to web-based
information integration. International Journal of
Cooperative Information Systems, 2001.
[13] N. Kushmerick, D. Weld, and R. Doorenbos. Wrapper
induction for information extraction. In Proceedings of
the International Joint Conference on Artificial
Intelligence (IJCAI), 1997., 1997.
[14] Thomas Leonard and Hugh Glaser. Large scale
acquisition and maintenance from the web without
source access. In Siegfried Handschuh, Rose
Dieng-Kuntz, and Steffen Staab, editors, Proceedings
Workshop 4, Knowledge Markup and Semantic
Annotation, K-CAP 2001, 2001.