Technology. His Current Research Interests In- Clude Neural Computation, Statistical Learning Theory, and Handwriting Recognition. Kwok and Yeung: Constructive Algorithms for Structure Learning \some Approximation Properties of Projection Pursuit Learning Networks," in Advances in Neural Information

15 63] L.K. Jones, \A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training," The Annals of Statistics,cursive dynamic node creation in multilayer neural networks," \Use of a quasi-Newton method in a feedforward neural network construction algorithm," IEEE Wahba, \Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross-validation," Numerische Mathe-13 vergence is also an important theoretical yet crucial practical issue. Recent results 62], 63], 131], 132], 133] have shown that, under certain regularity conditions, the approximation error typically improves as O(1 n), where n is the number of hidden units in the network. However, as mentioned in Section II-D.3, some of these results are not applicable to constructive algorithms when a greedy approach is taken, whereas others are applicable to greedy algorithms but with the detailed conditions diierent from those used in practice. For example, in both 62] and 63], the iterative sequence of network estimates is formed from a convex combination of the previous network function f n?1 and the new hidden unit function g n , (7) where 0 n 1. In algorithms like the cascade-correlation algorithm (Section IV-C), however, the new f n is formed from full linear combination of the old and new hidden unit functions. The weights connecting the old hidden units to the output unit are thus not constrained as a group, as in (7). Besides, in (7), n must be learned together with the new hidden unit g n , while in the cascade-correlation algorithm, the parameters of the new hidden unit are rst learned and then the output layer weights. Moreover, the objective function to be minimized in 62], 63] is diierent from that in the cascade-correlation algorithm. Modifying and applying these useful theoretical results to the analysis of diierent constructive algorithms is thus beneecial. Some initial progress has been reported in 119], 120], 121]. As mentioned in Section II-C, one must decide when to stop the constructive algorithm. This is important to achieve a proper bias-variance trade-oo. A good stopping criterion will be one based on an estimate of the generalization performance of the network. But computing an accurate estimation eeciently for neural networks is not an easy problem to solve, and this will still be an important research topic. Up to now, a comprehensive performance comparison of diierent constructive algorithms is …