Sir Arthur - Learning to walk on six Legs
by Frank Kirchner
The objective of the Sir Arthur project is to demonstrate the application
of Reinforcement Learning to a multi-degree of freedom walking machine.
It is shown how learning is used at the elementary level for basic swing
and stance motion, required to achieve the simple walking of one leg. Following
this, a more complex, higher level behaviour is learned involving the coordination
of each leg to achieve an overall forward motion. Finally, these behaviours
are applied to a goal orientated objective where Sir Arthur is required
to find areas of maximum light intensity.
16 motors are used to provide swing and stance for each of the six legs
and two degrees of freedom for each of the two joints. This allows Sir
Arthur to lift its head and look around to re-orientate relative to a light
source. The forward movement on six-leg machines is achieved by swinging
three legs to the front while at the same time performing a stance movement
on the others.
Sir Arthur is build in three segments: Each segment comprises two legs.
The segments are connected by two actively controllable degree-of-freedom
joints. The joints enable Sir Arthur to raise and pivot the front and backsegments,
thereby allowing for 3D movements. On each segment, there is a microprocessor
to perform the basic control tasks. An additional microcontroller, the
master controller is installed in the middle segment to perform higher
level processing tasks. The master and the slave controller use a bidirectional
8 bit parallel link for communication. The master sends commands to the
slaves to perform elementary swing and stance behaviours while it receives
status information and global sensor data.
Reinforcement Learning was implemented to learn a trajectory in the
state space that is formed by the possible positions of the legs. The reinforcement
to the robot is computed from the difference between a reference trajectory
and the currently performed movement. Using this reinforcement, the system
learns to perform the correct sequence of actions to swing and stance one
leg.
More difficult to learn is the coordination of the individual legs to
achieve a more complex behaviour such as walking forward. In this case,
Reinforcement Learning was also used to accomplish this task. A technique
called Experience Replay was used to speed up the learning. Experience
Replay stores successful sequences and uses them as a bias during later
learning stages.
After long sequences of unsuccessful trials, first signs of coordinated,
though still wrong, behaviour can be observed. As coordination increases,
the robot performs more and more forward steps until the correct coordination
scheme is finally found.
The most interesting applications for autonomous mobile robots are those
which require a high kinematic complexity of the robot. Examples are the
exploration of remote planets, the inspection of rough terrain, destroyed
or contaminated areas. However, the complex kinematics of the robot require
methods to control the system which are not offered by classic control
theory. Methods provided by Artificial Intelligence are not only useful
to solve such problems; currently, they are the only ones which can be
applied successfully. Information on Sir Arthur on the Web at: http://www.gmd.de/People/Frank.Kirchner/sir.arthur.html
Please contact:
Frank Kirchner - GMD
Tel: +49 2241 14 2402
E-mail: frank.kirchner@gmd.de
