Networks for approximation and learning

The problem of the approximation of nonlinear mapping, (especially continuous mappings) is considered. Regularization theory and a theoretical framework for approximation (based on regularization techniques) that leads to a class of three-layer networks called regularization networks are discussed. Regularization networks are mathematically related to the radial basis functions, mainly used for strict interpolation tasks. Learning as approximation and learning as hypersurface reconstruction are discussed. Two extensions of the regularization approach are presented, along with the approach's corrections to splines, regularization, Bayes formulation, and clustering. The theory of regularization networks is generalized to a formulation that includes task-dependent clustering and dimensionality reduction. Applications of regularization networks are discussed. >

[1]  I. J. Schoenberg,et al.  Metric spaces and positive definite functions , 1938 .

[2]  A Tikhonov,et al.  Solution of Incorrectly Formulated Problems and the Regularization Method , 1963 .

[3]  Amiel Feinstein,et al.  Applications of harmonic analysis , 1964 .

[4]  S. G. Mikhlin,et al.  The problem of the minimum of a quadratic functional , 1965 .

[5]  藤田 宏.  S.G. Mikhlin: The Problem of the Minimum of a Quadratic Functional, Holden-Day INC., San Francisco, 1965, 155+ix頁, 18×25cm, 3,580円. , 1965 .

[6]  L. Goddard.  Approximation of Functions , 1965, Nature.

[7]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[8]  D. Marr.  A theory of cerebellar cortex , 1969, The Journal of physiology.

[9]  R. L. Hardy.  Multiquadric equations of topography and other irregular surfaces , 1971 .

[10]  J. Albus.  A Theory of Cerebellar Function , 1971 .

[11]  R. N. Desmarais,et al.  Interpolation using surface splines. , 1972 .

[12]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[13]  H. L. Resnikoff.  On the psychophysical function , 1975 .

[14]  J. Stewart.  Positive definite functions and generalizations, an historical survey , 1976 .

[15]  Jean Duchon,et al.  Splines minimizing rotation-invariant semi-norms in Sobolev spaces , 1976, Constructive Theory of Functions of Several Variables.

[16]  G. Wahba.  Practical Approximate Solutions to Linear Operator Equations When the Data are Noisy , 1977 .

[17]  Ray J. Solomonoff,et al.  Complexity-based induction systems: Comparisons and convergence theorems , 1978, IEEE Trans. Inf. Theory.

[18]  Peter Craven,et al.  Smoothing noisy data with spline functions , 1978 .

[19]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[20]  J. Meinguet.  Multivariate interpolation at arbitrary points made simple , 1979 .

[21]  Michael Brady.  MIT Progress in Understanding Images , 1982 .

[22]  C. J. Stone,et al.  Optimal Global Rates of Convergence for Nonparametric Regression , 1982 .

[23]  R. Franke.  Scattered data interpolation: tests of some methods , 1982 .

[24]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[25]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[26]  C. J. Stone,et al.  Additive Regression and Other Nonparametric Models , 1985 .

[27]  T. Flash,et al.  The coordination of arm movements: an experimentally confirmed mathematical model , 1985, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[28]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[29]  David L. Waltz,et al.  Toward memory-based reasoning , 1986, CACM.

[30]  C. Micchelli.  Interpolation of scattered data: Distance matrices and conditionally positive definite functions , 1986 .

[31]  Nira Dyn,et al.  Interpolation of scattered Data by radial Functions , 1987, Topics in Multivariate Approximation.

[32]  A. Lapedes,et al.  Nonlinear signal processing using neural networks: Prediction and system modelling , 1987 .

[33]  Tomaso Poggio,et al.  Probabilistic Solution of Ill-Posed Problems in Computational Vision , 1987 .

[34]  Stephen M. Omohundro,et al.  Efficient Algorithms with Neural Network Behavior , 1987, Complex Syst..

[35]  Eric B. Baum,et al.  On the capabilities of multilayer perceptrons , 1988, J. Complex..

[36]  M. Bertero,et al.  Ill-posed problems in early vision , 1988, Proc. IEEE.

[37]  Alan L. Yuille,et al.  The Motion Coherence Theory , 1988, [1988 Proceedings] Second International Conference on Computer Vision.

[38]  John G. Harris,et al.  An Analog VLSI Chip for Thin-Plate Surface Interpolation , 1988, NIPS.

[39]  J. Keeler.  Comparison Between Kanerva's SDM and Hopfield-Type Neural Networks , 1988, Cogn. Sci..

[40]  David S. Broomhead,et al.  Multivariable Functional Interpolation and Adaptive Networks , 1988, Complex Syst..

[41]  Tomaso A. Poggio,et al.  Representation properties of multilayer feedforward networks , 1988, Neural Networks.

[42]  Edward J Wegman,et al.  Symposium on the Interface: Computing Science and Statistics (20th). Theme: Computationally Intensive Methods in Statistics Held in Reston, Virginia on April 20-23, 1988 , 1988 .

[43]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[44]  W. Madych,et al.  Multivariate interpolation and condi-tionally positive definite functions , 1988 .

[45]  Pentti Kanerva,et al.  Sparse Distributed Memory , 1988 .

[46]  J. Doyne Farmer,et al.  Exploiting Chaos to Predict the Future and Reduce Noise , 1989 .

[47]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[48]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[49]  S. Renals,et al.  Phoneme classification experiments using radial basis functions , 1989, International 1989 Joint Conference on Neural Networks.

[50]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[51]  Tomaso A. Poggio,et al.  Representation Properties of Networks: Kolmogorov's Theorem Is Irrelevant , 1989, Neural Computation.

[52]  John Y. Aloimonos,et al.  Unification and integration of visual modules: an extension of the Marr Paradigm , 1989 .

[53]  Martin Casdagli,et al.  Nonlinear prediction of chaotic time series , 1989 .

[54]  H. White,et al.  Universal approximation using feedforward networks with non-sigmoid hidden layer activation functions , 1989, International 1989 Joint Conference on Neural Networks.

[55]  E. Kansa.  Multiquadrics—A scattered data approximation scheme with applications to computational fluid-dynamics—I surface approximations and partial derivative estimates , 1990 .

[56]  T. Poggio,et al.  A network that learns to recognize three-dimensional objects , 1990, Nature.

[57]  Tomaso A. Poggio,et al.  Extensions of a Theory of Networks for Approximation and Learning , 1990, NIPS.

[58]  Federico Girosi,et al.  Parallel and deterministic algorithms from MRFs: surface reconstruction and integration , 1990, ECCV.

[59]  L. Galway.  Spline Models for Observational Data , 1991 .

[60]  V. Tikhomirov.  On the Representation of Continuous Functions of Several Variables as Superpositions of Continuous Functions of one Variable and Addition , 1991 .

[61]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[62]  Shimon Edelman,et al.  Bringing the Grandmother back into the Picture: A Memory-Based View of Object Recognition , 1990, Int. J. Pattern Recognit. Artif. Intell..

[63]  I. Omiaj,et al.  Extensions of a Theory of Networks for Approximation and Learning : dimensionality reduction and clustering , 2022 .