Iterative design of regularizers based on data by minimizing generalization errors

In our previous study (1998) we proposed a theoretical evaluation of generalization errors. However, it suffered from from serious difficulties: 1) it assumes that true model parameters and noise variance are known a priori; and 2) it assumes that input variables are mutually independent. These assumptions prevent its application to real data. The present paper succeeds in overcoming these two difficulties. A key idea is to iteratively estimate these parameters and generalization errors from data. Introducing correlations between input variables is not intrinsically difficult, although it makes computation much more complex than the cases where input variables are mutually independent.

[1]  Masumi Ishikawa,et al.  Rule extraction by successive regularization , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[2]  Kazumi Saito,et al.  Second-Order Learning Algorithm with Squared Penalty Term , 1996, Neural Computation.

[3]  Carl E. Rasmussen,et al.  Pruning from Adaptive Regularization , 1994, Neural Computation.

[4]  Masumi Ishikawa,et al.  Designing regularizers by minimizing generalization errors , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[5]  Lars Kai Hansen,et al.  Regularization with a Pruning Prior , 1997, Neural Networks.

[6]  Masumi Ishikawa,et al.  Structural learning with forgetting , 1996, Neural Networks.