Combining Exploratory Projection Pursuit and Projection Pursuit Regression with Application to Neural Networks

We present a novel classification and regression method that combines exploratory projection pursuit (unsupervised training) with projection pursuit regression (supervised training), to yield a new family of cost/complexity penalty terms. Some improved generalization properties are demonstrated on real-world problems.

[1]  R. Fisher.  THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS , 1936 .

[2]  George S. Sebestyen,et al.  Decision-making processes in pattern recognition , 1962 .

[3]  George S Sebestyen,et al.  Decision-making processes in pattern recognition (ACM monograph series) , 1962 .

[4]  E. B. Andersen,et al.  Modern factor analysis , 1961 .

[5]  J. Kruskal.  TOWARD A PRACTICAL METHOD WHICH HELPS UNCOVER THE STRUCTURE OF A SET OF MULTIVARIATE OBSERVATIONS BY FINDING THE LINEAR TRANSFORMATION WHICH OPTIMIZES A NEW “INDEX OF CONDENSATION” , 1969 .

[6]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[7]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.

[8]  J. Friedman,et al.  Projection Pursuit Regression , 1981 .

[9]  Richard F. Lyon,et al.  A computational model of filtering, detection, and compression in the cochlea , 1982, ICASSP.

[10]  E. Bienenstock,et al.  Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex , 1982, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[11]  D. Freedman,et al.  Asymptotics of Graphical Projection Pursuit , 1984 .

[12]  Ramesh C. Jain,et al.  Three-dimensional object recognition , 1985, CSUR.

[13]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[14]  Geoffrey E. Hinton,et al.  Experiments on Learning by Back Propagation. , 1986 .

[15]  Robin Sibson,et al.  What is projection pursuit , 1987 .

[16]  J. Friedman.  Exploratory Projection Pursuit , 1987 .

[17]  P. Hall.  Estimating the direction in which a data set is most interesting , 1988 .

[18]  Richard Lippmann,et al.  Review of Neural Networks for Speech Recognition , 1989, Neural Computation.

[19]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[20]  P. Hall.  On Polynomial-Based Projection Indices for Exploratory Projection Pursuit , 1989 .

[21]  A. Barron,et al.  Statistical properties of artificial neural networks , 1989, Proceedings of the 28th IEEE Conference on Decision and Control,.

[22]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[23]  S. M. Carroll,et al.  Construction of neural nets using the radon transform , 1989, International 1989 Joint Conference on Neural Networks.

[24]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[25]  Josef Skrzypek,et al.  Synergy of Clustering Multiple Back Propagation Networks , 1989, NIPS.

[26]  Peter Seitz,et al.  Minimum class entropy: A maximum information approach to layered networks , 1989, Neural Networks.

[27]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[28]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[29]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[30]  Robert Hecht-Nielsen,et al.  Theory of the backpropagation neural network , 1989, International 1989 Joint Conference on Neural Networks.

[31]  Michael C. Mozer,et al.  Using Relevance to Reduce Network Size Automatically , 1989 .

[32]  Kurt Hornik,et al.  Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks , 1990, Neural Networks.

[33]  G. Wahba.  Spline models for observational data , 1990 .

[34]  Halbert White,et al.  Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings , 1990, Neural Networks.

[35]  Barak A. Pearlmutter,et al.  Chaitin-Kolmogorov Complexity and Generalization in Neural Networks , 1990, NIPS.

[36]  David E. Rumelhart,et al.  Generalization by Weight-Elimination with Application to Forecasting , 1990, NIPS.

[37]  David J. C. MacKay,et al.  Unsupervised Classifiers, Mutual Information and 'Phantom Targets' , 1991, NIPS.

[38]  Nathan Intrator,et al.  Supervised and unsupervised feature extraction from a cochlear model for speech recognition , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.

[39]  Yoshifusa Ito,et al.  Representation of functions by superpositions of a step or sigmoid function and their applications to neural network theory , 1991, Neural Networks.

[40]  Josh I. GoldCenter.  Three-Dimensional Object Recognition Using an Unsupervised Neural Network: Understanding the Distinguishing Features , 1991 .

[41]  John Moody,et al.  Note on generalization, regularization and architecture selection in nonlinear learning systems , 1991, Neural Networks for Signal Processing Proceedings of the 1991 IEEE Workshop.

[42]  L. Cooper,et al.  Synaptic plasticity in visual cortex: comparison of theory with experiment. , 1991, Journal of neurophysiology.

[43]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[44]  J. Sklansky,et al.  Robust Classifiers by Mixed Adaptation , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Kurt Hornik,et al.  Approximation capabilities of multilayer feedforward networks , 1991, Neural Networks.

[46]  Robert Azencott,et al.  Image compression with back propagation: Improvement of the visual restoration using different cost functions , 1991, Neural Networks.

[47]  Nathan Intrator,et al.  Feature Extraction Using an Unsupervised Neural Network , 1992, Neural Computation.

[48]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[49]  Nathan Intrator,et al.  Objective function formulation of the BCM theory of visual cortical plasticity: Statistical connections, stability conditions , 1992, Neural Networks.

[50]  Geoffrey E. Hinton,et al.  Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[51]  Nathan Intrator,et al.  Phonetic classification of timit segments preprocessed with lyon's cochlear model using a supervised/unsupervised hybrid neural network , 1992, ICSLP.

[52]  Nathan Intrator,et al.  Three-Dimensional Object Recognition Using an Unsupervised BCM Network: The Usefulness of Distinguishing Features , 1993, Neural Computation.

[53]  David J. Field,et al.  What Is the Goal of Sensory Coding? , 1994, Neural Computation.

[54]  S. Klinke,et al.  Exploratory Projection Pursuit , 1995 .