CharToon - A JAVA-based Animation System
by Paul ten Hagen
The 3-year project FASE (Facial Analysis and Synthesis of Expressions),
started in spring 1997, has entered its second phase. The aim
is to develop a system which can recognize facial movements in
front of a camera and use the data thus produced to animate models
of the human face. Basic editing tools have been implemented in
the CharToon system, consisting of a collection of JAVA programs
enabling the interactive construction of parametrized 2D drawings
and a set of time curves to animate the drawings. The project
is a joint effort of Delft University of Technology (TUD) and
CWI and is sponsored by the Dutch Technology Foundation STW.
Previous work, including CWI’s FERSA project in the early 1990?s
on automatic real-time lip-synchronous speech animation, had shown
that two technologies play a pivotal role to realize such a system:
Facial Analysis (FA) and Facial Synthesis (FS). An important aspect
of FASE is the integration of both technologies, in order to achieve
life-likeness of the estimated facial information and motion.
The FA module, covered by TUD, receives as input video images
of a face (the ‘user’) and possibly additional information (eg,
text and audio sources), on the basis of which the parameters
describing the facial movements and expressions are estimated.
The FS module, developed at CWI, contains a generic facial model
and an animation model. It uses the estimated facial parameters
as an input to deform and move the facial model and thus ‘mimic’
the looks and movements of the user. The results are used as a
feedback input for FA. In addition, a third module, ‘Model Editing
and Parameter Control Interface’ not only facilitates the development
and set up of the (face) models and the generation of new animation
models, but also provides easy control over the whole process
of changing facial parameters – a highly desirable feature, in
view of the close interplay between analysis and synthesis. The
integration of FA and FS is also done at CWI.
The FS module allows two kinds of facial models:
- 3D physically based ones, aiming at realism
- 2D drawings, aiming at cartoon-like effects, stylized avatar faces
or possibly also animation of 2D photographs of a real face.
For the 2D models we developed a Face Editor to design the drawing
to be animated, an Animation Editor to specify the time-behaviour
of the drawing’s animation parameters, leading to a movie script,
and a Face Player to animate the drawing using this script. These
three tools are integrated in the CharToon system. The implementation
in JAVA makes the system suitable for Web applications.
The hierachical drawing program FaceEditor consists of basic building
blocks like polygons and ellipses, objects composed of animated
polygons and other shape-changing objects, and ready-made complex
objects like an eye with pupil and eyelids.With AnimEditor values
of animation parameters for computer facial (or other) models
can be graphically specified and modified. The editor operates
on a window resembling a musical score, with a ‘staff’? for each
parameter and dots on the staff indicating the parameter?s behaviour
in time. There is a facility to switch on and off audio channels.
AnimEditor is also able to perform constraint management. Finally,
in combination with the tool PERSONA, AnimEditor is also used
to build realistic 3D facial models on the basis of the input
facial parameters. Movies with sound, as constructed by AnimEditor
and FaceEditor, can be played by FacePlayer. For small drawings,
one can (just) achieve a rate of 20 frames/sec on an otherwise
empty Sun Sparc Ultra. Different parts of a drawing can be controlled
in parallel.
Current research at CWI concentrates on:
- gathering performer data to distillate generic and person-dependent
characteristics of certain facial expressions
- fine-tuning the muscle model, add features to and improve performance
of the physically-based model
- developing an interactive conformation tool
- extending the animation editing tool with mechanisms to allow
the definition and dynamic use of animations as building blocks,
synchronisation and other constraints
- trying out the existing tools in different applications.
The area of possible applications is very wide. The television
(production and post-production) and telecommunication (eg, low
bandwidth communication such as in videoconferencing and videophoning)
industries are interested in realistic 3D rendering of facial
expressions (efficient image coding). 2D caricature rendering
is of interest for the film industry (animation) and in man-machine
interaction (social user interfaces: avatars). Other applications
include: entertainment (games, interactive television, synthesized
actors), lip synchronisation (text or voice based, possibly also
adjusted for the deaf), and facial surgical planning and simulation.
Several potential industrial users are represented in the user
group accompanying the FASE project (a standard procedure in STW
projects): NOB (major Dutch television production company), KPN
Research (communication systems), Institute for Phonetical Science
(University of Amsterdam), Institute for the Deaf (Sint-Michielsgestel),
and General Design (a CWI spin-off company, eg, for the design
of Web sites). The CharToon system will be available for researchers
free of charge by the end of the year. Information on the Web:
http://www.cwi.nl/FASE
Please contact:
Paul ten Hagen - CWI
Tel: +31 20 592 4133
E-mail: Paul.ten.Hagen@cwi.nl
