Uncertainty Bounds in Parameter Estimation with Limited Data

Consider the problem of determining uncertainty bounds for parameter estimates with a small sample size of data. Calculating uncertainty bounds requires information about the distribution of the estimate. Although many common parameter estimation methods (e.g., maximum likelihood, least squares, maximum a posteriori, etc.) have an asymptotic normal distribution, very little is usually known about the finite-sample distribution. This paper presents a method for characterizing the distribution of an estimate when the sample size is small. The approach works by comparing the actual (unknown) distribution of the estimate with an “idealized” (known) distribution. Some discussion and analysis are included that compare the approach here with the well-known bootstrap and saddlepoint methods. Example applications of the approach are presented in the areas of signal-plus-noise modeling, nonlinear regression, and time series correlation analysis. The signal-plus-noise problem is treated in greatest deta il; this problem arises in many contexts, including state-space modeling, the problem of combining several independent estimates, and quantile calculation for projectile accuracy analysis.

[1]  Saddlepoint Approximations in the Case of Intractable Cumulant Generating Functions , 1991 .

[2]  Andeej Pizman,et al.  Invited discussion paper small-sample distributional properties of nonlinear regression estimators (a geometric approach) , 1990 .

[3]  Nancy Reid,et al.  Saddlepoint Methods and Statistical Inference , 1988 .

[4]  Elvezio Ronchetti,et al.  Small Sample Asymptotics , 1990 .

[5]  Calyampudi R. Rao,et al.  Linear Statistical Inference and Its Applications. , 1975 .

[6]  T. Anderson.  Statistical analysis of time series , 1974 .

[7]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[8]  L. Ljung.  Convergence analysis of parametric identification methods , 1978 .

[9]  Matrix Weighting of Several Regression Coefficient Vectors , 1993 .

[10]  A. Chesher.  The effect of measurement error , 1991 .

[11]  D. P. Gaver,et al.  Combining Information: Statistical Issues and Opportunities for Research , 1993 .

[12]  D. A. Shnidman,et al.  Efficient computation of the circular error probability (CEP) integral , 1995, IEEE Trans. Autom. Control..

[13]  S. Huzurbazar.  Practical Saddlepoint Approximations , 1999 .

[14]  Malay Ghosh,et al.  Small Area Estimation: An Appraisal , 1994 .

[15]  B. Hoadley.  Asymptotic Properties of Maximum Likelihood Estimators for the Independent Not Identically Distributed Case , 1971 .

[16]  James C. Spall,et al.  First-order data sensitivity measures with applications to a multivariate signal-plus-noise problem , 1990 .

[17]  Jayanta K. Ghosh,et al.  Higher Order Asymptotics , 1994 .

[18]  S. Bisgaard,et al.  Standard errors for the eigenvalues in second-order response surface models , 1996 .

[19]  J. Berger,et al.  Empirical Bayes Estimation of Rates in Longitudinal Studies , 1983 .

[20]  Anthony C. Davison,et al.  Saddlepoint approximations in resampling methods , 1988 .

[21]  Calyampudi Radhakrishna Rao,et al.  Linear Statistical Inference and its Applications , 1967 .

[22]  J. L. Maryak,et al.  A Feasible Bayesian Estimator of Quantiles for Projectile Accuracy From Non-iid Data , 1992 .

[23]  Silvelyn Zwanzig,et al.  Discussion to A. Pazman: Small-sample distributional properties of nonlinear regression estimator (a geometric approach) , 1990 .

[24]  S. Yakowitz,et al.  Error inference for nonparametric regression , 1991 .

[25]  Eric Walter,et al.  Qualitative and quantitative experiment design for phenomenological models - A survey , 1990, Autom..

[26]  Jack Kaplan,et al.  Estimators for the One-Way Random Effects Model with Unequal Error Variances , 1981 .

[27]  George Casella,et al.  Explaining the Saddlepoint Approximation , 1999 .

[28]  On the Approximation of Saddlepoint Expansions in Statistics , 1994, Econometric Theory.

[29]  Jan Kmenta,et al.  Elements of econometrics , 1988 .

[30]  R. Shumway,et al.  Estimation and tests of hypotheses for the initial mean and covariance in the kalman filter model , 1981 .

[31]  Robert Tibshirani,et al.  Computer‐Intensive Statistical Methods , 2006 .

[32]  Fang-Kuo Sun,et al.  A maximum likelihood algorithm for the mean and covariance of nonidentically distributed observations , 1982 .

[33]  J. Spall.  Uncertainty bounds for parameter identification with small sample sizes , 1995, Proceedings of 1995 34th IEEE Conference on Decision and Control.

[34]  R. Laha.  Probability Theory , 1979 .

[35]  C. Lunneborg.  Data Analysis by Resampling: Concepts and Applications , 1999 .

[36]  Robert L. Taylor,et al.  Selected proceedings of the Sheffield Symposium on Applied Probability , 1991 .

[37]  Hanif D. Sherali,et al.  An enhanced RSM algorithm using gradient-deflection and second-order search strategies , 1994, Proceedings of Winter Simulation Conference.

[38]  Richard H. Smith.  Maximum likelihood mean and covariance matrix estimation constrained to general positive semi-definiteness , 1985 .

[39]  M. E. Welch,et al.  Bayesian analysis of time series and dynamic models , 1990 .

[40]  R. Serfling.  Approximation Theorems of Mathematical Statistics , 1980 .

[41]  H. Daniels.  Saddlepoint Approximations in Statistics , 1954 .