A Connection Between GRBF and MLP

Both multilayer perceptrons (MLP) and Generalized Radial Basis Functions GRBF) have good approximation properties, theoretically and experimentally. Are they related? The main point of this paper 'is to show that for normalized inputs, multilayer perceptron networks are radial function networks albeit wth a non-standard radial function). This provides an interpretation of the weights u7 as centers t of the radial function network, and therefore as equivalent to templates. This 'Insight may be useful for practical applications, ncluding better 'Initialization procedures for MLP. In the remainder of the paper, we discuss the relation between the radial functions that correspond to the sigmoid for normalized inputs and well-behaved radial basis functions, such as the Gaussian. In particular, we observe that the radial function associated with the sigmoid 'is an activation function that is good approximation to Gaussian basis functions for a range of values of the bias parameter. The mplication is that a MLP network can always simulate a Gaussian GRBF network (with the same nmber of uits but less parameters); the converse is true oly for certain values of the bias parameter. Numerical experiments indicate that this constraint 'is not always satisfied in practice by MLP networks trained with backpropagation. Multiscale RBF networks, on the other hand, can approximate MLP networks with a smilar number of parameters.