LARGE SCALE SIMULATIONS FOR LEARNING CURVES

The universal asymptotic scaling laws proposed by Amari et al. 2,11 are studied in large scale simulations using a CM5. Small stochastic feed-forward networks trained with back-propagation and conjugate gradient descent are investigated. In the range of a large number of training patterns t, the predicted asymptotic 1/t scaling is observed. For a medium range t a faster scaling in the number of training patterns t than 1/t is observed. This effect is explained by using higher order corrections of the likelihood expansion. For small t it is shown, that the scaling law changes drastically, when the network undergoes a transition from permutation symmetric to permutation symmetry broken phase. This effect is related to previous theoretical work .