Object-based Manipulation Tools for Postprocessing Video Applications
by Henri Nicolas
In the framework of the ESPRIT project NEMESIS (NEw MultimEdia Services
using analysIs Synthesis), partly carried out by the Temis research group
at IRISA/INRIA, Rennes, automatic or semi-automatic tools are being developed
which enable object-based manipulations of the original image sequences
for post-processing video manipulations.
When a video sequence is acquired by a moving camera, it contains generally
both moving objects and a static background. By computing an object-based
segmentation of the scene for each image, the background can be separated
from the rest of the scene in order to create the mosaic image. This global
information can therefore be used for video editing (for example suppression
of an object), or to virtually change the camera point of view, ie, the
camera location and its focal. This leads to the creation of video sequences
which have never been acquired by a camera, but which are obtained using
real data provided by the real camera shot. This kind of automatic tools
is of great interest for the post-production world in which, with the existing
techniques, long and tedious manual efforts are necessary to achieve such
manipulations.
In practice, once the images have been acquired by the camera, a human
operator defines the manipulations which must be done on the objects existing
in the original data. Furthermore, in order to obtain very high qualities
on the computed image sequences, human intervention may be required to
correctly set-up the algorithms according to the image characteristics.
Nevertheless, the use of the proposed approach in a purely automatic way
is possible (see the illustrations below) but lower image quality can be
expected.
Different techniques and algorithms have been developed in order to
design such manipulation tools. In practice, an artificial image sequence
is created using the four following steps:
- Camera motion estimation.
Precise camera estimation is required in order to create the mosaic
image. This is obtained by using a three parameters camera motion model,
ie two translational and one zoom motion parameters. For each image, these
parameters are estimated using a temporal linking technique combined with
a comparison with the currently obtained mosaic image.
- Object-based image segmentation.
The creation of the mosaic image requires an object-based segmentation
for each original image in order to differentiate the background from the
other objects. For the first image, a human operator has to define the
objects which must not be included in the mosaic (this may include static
objects which cannot be automatically separated from the background by
motion estimation). For the other images, an object-based motion estimation
technique combines with the current mosaic image permits to select the
information which can be used to update the current mosaic.
- Creation of a mosaic image.
The mosaic is defined as the global view of the scene background resulting
from an image sequence obtained by a moving camera. For each image, this
mosaic image is created or updated using the current initial image and
taking account the camera motion estimation and the object-based segmentation
previously estimated.
- Creation of the artificial image sequence.
A human operator specifies the treatment which must be done on the images
(for example, positioning of a given object in the center of the images).
The artificial sequence is therefore computed using the original images,
the segmentation and the mosaic images.
First versions of these tools have already been developed which modify
artificially the camera point of view or eliminate a given object from
the scene using camera motion estimation, object-based segmentation and
mosaic images. Nevertheless, new techniques continue to be developed in
order to improve the quality of the results. Finally, a major perspective
of this work is related to the mixing of synthetic objects together with
real data.
Please contact:
Henri Nicolas - IRISA/INRIA
Tel: +33 2 99 84 74 25
E-mail: hnicolas@irisa.fr
