An efficient calculation of Fisher information matrix: Monte Carlo approach using prior information

The Fisher information matrix (FIM) is a critical quantity in several aspects of system identification, including input selection and confidence region calculation. Analytical determination of the FIM in a general system identification setting may be difficult or almost impossible due to intractable modeling requirements and/or high-dimensional integration. A Monte Carlo (MC) simulation-based technique was introduced by the second author to address these difficulties (Spall, 2005). This paper proposes an extension of the MC algorithm in order to enhance the statistical qualities of the estimator of the FIM. This modified MC algorithm is particularly useful in those cases where the FIM has a structure with some elements being analytically known from prior information and the others being unknown. The estimator of the FIM, obtained by using the proposed MC algorithm, simultaneously preserves the analytically known elements and reduces the variances of the estimators of the unknown elements by capitalizing on the information contained in the known elements.

[1]  P. Bickel,et al.  Mathematical Statistics: Basic Ideas and Selected Topics , 1977 .

[2]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[3]  Max Tegmark,et al.  Karhunen-Loève Eigenvalue Problems in Cosmology: How Should We Tackle Large Data Sets? , 1996, astro-ph/9603021.

[4]  J. Spall.  Multivariate stochastic approximation using a simultaneous perturbation gradient approximation , 1992 .

[5]  George Chryssolouris,et al.  Confidence interval prediction for neural network models , 1996, IEEE Trans. Neural Networks.

[6]  Daniel C. Kammer.  Sensor set expansion for modal vibration testing , 2005 .

[7]  James C. Spall,et al.  Introduction to Stochastic Search and Optimization. Estimation, Simulation, and Control (Spall, J.C. , 2007 .

[8]  B. Efron,et al.  Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information , 1978 .

[9]  Roger Ghanem,et al.  Asymptotic Sampling Distribution for Polynomial Chaos Representation of Data: A Maximum Entropy and Fisher information approach , 2006, CDC.

[10]  Sonjoy Das.  EFFICIENT CALCULATION OF FISHER , 2007 .

[11]  James C. Spall,et al.  Adaptive stochastic approximation by the simultaneous perturbation method , 2000, IEEE Trans. Autom. Control..

[12]  J. Spall.  Feedback and weighting mechanisms for improving Jacobian (Hessian) estimates in the adaptive simultaneous perturbation algorithm , 2006, 2006 American Control Conference.

[13]  J. Spall.  Monte Carlo Computation of the Fisher Information Matrix in Nonstandard Settings , 2005 .

[14]  J. T. Hwang,et al.  Prediction Intervals for Artificial Neural Networks , 1997 .

[15]  C. N Bouza,et al.  Spall, J.C. Introduction to stochastic search and optimization. Estimation, simulation and control. Wiley Interscience Series in Discrete Mathematics and Optimization, 2003 , 2004 .

[16]  H. Jeffreys.  An invariant form for the prior probability in estimation problems , 1946, Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences.

[17]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.