Contrastive Indexing of Full Text Documents
by Laurent Romary and Patrice Bonhomme
In the context of the general Aquarelle scenario, the creation of
folders allows a user to put together pieces of information which he considers
useful for his own purpose. In particular, he may include textual fields
which in turn have to be made accessible for further retrieval. To this
end, we designed a full text indexing method which, rather than providing
an absolute set of indexes for each textual field, aims at contrasting
each of them to the other fields the user might point to either in the
same folder or within other folders he has created or extracted from an
Aquarelle server.
The basic idea behind the contrastive indexing method is to consider
a given document or rather the set of tokens it contains as a sample taken
from the set of all tokens belonging to the reference corpus of documents
it belongs to. The frequency of the token within the document can then
be compared to the expected distribution computed from the reference corpus,
in order to evaluate whether it is inkeeping with it, or on the contrary
too far from it not to be interpreted as indicating a particular relevance
for the document. For each document, we thus compute a set of so-called
contrasting tokens which is a good indication of its informational content
relatively to the contents of the documents it is compared to. As a consequence,
this method has different interesting properties which both from a linguistic
and information retrieval point of view makes it a good option for an optimal
full text indexing mechanism:
- there is no need for a specific list of grammatical words (ie stop-list)
which have to be avoided during the indexing process, since these are generally
subject to a uniform distribution among a set of documents taken from the
same field or belonging to the same textual genre (eg historical descriptions,
newspaper articles etc)
- furthermore, not only grammatical words are being dropped from the
candidate list of indexing terms, but also those words which, although
being meaningful from an absolute point of view, are uniformly represented
within the reference set of documents and thus are not relevant for the
description of any specific one. For instance, within a set of documents
describing historical buildings words like architecture or architectural
are not likely to appear as contrasting tokens:
- as a consequence, the method can be seen at least to a large
extent as being language independent, as it relies on a local model
of linguistic distribution. Still, even if this has not been specifically
tested, we might expect that for highly inflectional languages the result
might not be as good as those observed for French and English, unless a
lemmatizing phase is considered
- finally it can be observed that a given document can be indexed differently
according to the set of documents which contextualizes it, thus providing
a way to account for the differents viewpoints users might project onto
it when building up folders.
The full text indexing module has been considered as a semi-automatic
process provided to the user during the folder editing stage. As a matter
of fact the user always has the possibility to edit and validate the set
of candidate terms before these are actually inserted within the folder
itself.
Given the robustness of the method as we have observed it in our first
trials within the Aquarelle project, we have thought of extending it towards
a general mechanism of content identification within a set of more less
homogeneous documents. Indeed, what results from the contrastive indexing
process is a kind of thematic description of the document in comparison
with a given reference which acts as a background, hence the possibility
to iteratively group together documents with similar descriptions and further
to build up a thematic map of the whole reference database. This concept
has been recently applied within a project funded by the DGLF (Délégation
Générale à la Langue Française) aiming at automatically
producing thematic descriptions of a given web site. The contrastive indexing
method, combined with a hierarchical clustering algorithm has allowed us
to produce topic maps of a given web site independently of its actual language
or content domain.
Please contact:
Laurent Romary and Patrice Bonhomme - LORIA
Tel: +33 3 8359 2037
E-mail: {romary,bonhomme}@loria.fr
