Comparison between Expected and Observed Fisher Information in Interval Estimation

Maximum likelihood estimates and corresponding confidence regions of the estimates are commonly used in statistical inference. In practice, people usually construct approximate confidence regions with the Fisher information at given sample data based on the asymptotic normal distribution of the MLE (maximum likelihood estimator). Two common Fisher information matrices (FIMs, for multivariate parameters) are the observed FIM (the Hessian matrix of negative log-likelihood function) and the expected FIM (the expectation of the observed FIM). In this article, we prove that under certain conditions and with MSE (mean-squared error) criterion, approximate confidence interval of each element of the MLE with the expected FIM is at least as accurate as that with the observed FIM.

[1]  S. D. Hill,et al.  Least-informative Bayesian prior distributions for finite samples based on information theory , 1990 .

[2]  J. L. Maryak,et al.  A Feasible Bayesian Estimator of Quantiles for Projectile Accuracy From Non-iid Data , 1992 .

[3]  P. McCullagh.  Tensor Methods in Statistics , 1987 .

[4]  D. Freedman.  How Can the Score Test Be Inconsistent? , 2007 .

[5]  James C. Spall,et al.  Introduction to stochastic search and optimization - estimation, simulation, and control , 2003, Wiley-Interscience series in discrete mathematics and optimization.

[6]  James C. Spall,et al.  Relative performance of expected and observed fisher information in covariance estimation for maximum likelihood estimates , 2012, 2012 American Control Conference (ACC).

[7]  H. K. T. Ng,et al.  Parameter estimation for a modified Weibull distribution, for progressively type-II censored samples , 2005, IEEE Transactions on Reliability.

[8]  A. F. Smith,et al.  Statistical analysis of finite mixture distributions , 1986 .

[9]  B. Lindsay,et al.  On second-order optimality of the observed Fisher information , 1997 .

[10]  G. Molenberghs,et al.  What Can Go Wrong With the Score Test? , 2007 .

[11]  B. Efron,et al.  Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information , 1978 .

[12]  Anders E. Nordsjo.  Cramer-Rao bounds for a class of systems described by Wiener and Hammerstein models , 1997 .

[13]  James C. Spall,et al.  Confidence Intervals with Expected and Observed Fisher Information in the Scalar Case , 2020, 2020 American Control Conference (ACC).

[14]  N. Balakrishnan,et al.  Point and interval estimation for Gaussian distribution, based on progressively Type-II censored samples , 2003, IEEE Trans. Reliab..

[15]  Xiaodong Wang,et al.  Sequential Decentralized Parameter Estimation Under Randomly Observed Fisher Information , 2012, IEEE Transactions on Information Theory.

[16]  R. Shumway,et al.  Estimation and tests of hypotheses for the initial mean and covariance in the kalman filter model , 1981 .

[17]  Oleg Granichin.  The nonasymptotic confidence set for parameters of a linear control object under an arbitrary external disturbance , 2012 .

[18]  James C. Spall,et al.  Identification for Systems With Binary Subsystems , 2014, IEEE Transactions on Automatic Control.

[19]  R. Redner,et al.  Mixture densities, maximum likelihood, and the EM algorithm , 1984 .

[20]  Long Wang,et al.  Combining Subsystem and Full System Data with Application to Cold-Formed Steel Shear Wall , 2018, 2018 Annual American Control Conference (ACC).

[21]  James C. Spall,et al.  First-order data sensitivity measures with applications to a multivariate signal-plus-noise problem , 1990 .