Training feed-forward networks with the extended Kalman algorithm

It is shown that training feed-forward nets can be viewed as a system identification problem for a nonlinear dynamic system. For linear dynamic systems, the Kalman filter is known to produce an optimal estimator. Extended versions of the Kalman algorithm can be used to train feed-forward networks. The performance of the Kalman algorithm is examined using artificially constructed examples with two inputs, and it is found that the algorithm typically converges in a few iterations. Backpropagation is used on the same examples, and the Kalman algorithm invariably converges in fewer iterations. For the XOR problem, backpropagation fails to converge on any of the cases considered, whereas the Kalman algorithm is able to find solutions with the same network configurations.<<ETX>>