Splitting and merging components of a nonconjugate Dirichlet process mixture model

Abstract. The inferential problem of associating data to mixture components is difficult when components are nearby or overlapping. We introduce a new split-merge Markov chain Monte Carlo technique that efficiently classifies observations by splitting and merging mixture components of a nonconjugate Dirichlet process mixture model. Our method, which is a Metropolis-Hastings procedure with split-merge proposals, samples clusters of observations simultaneously rather than incrementally assigning observations to mixture components. Split-merge moves are produced by exploiting properties of a restricted Gibbs sampling scan. A simulation study compares the new split-merge technique to a nonconjugate version of Gibbs sampling and an incremental MetropolisHastings technique. The results demonstrate the improved performance of the new sampler.

[1]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[2]  P. Whittle ON STATIONARY PROCESSES IN THE PLANE , 1954 .

[3]  Robert W. Newcomb,et al.  On the simultaneous diagonalization of two semi-definite matrices , 1961 .

[4]  Alexander A. Lubischew On the Use of Discriminant Functions in Taxonomy , 1962 .

[5]  J. Berg,et al.  Factors influencing short and long term survival of breast cancer patients. , 1966, Surgery, gynecology & obstetrics.

[6]  W. K. Hastings,et al.  Monte Carlo Sampling Methods Using Markov Chains and Their Applications , 1970 .

[7]  H. Skipper,et al.  Kinetics of mammary tumor cell growth and implications for therapy , 1971, Cancer.

[8]  P. Jones Making Decisions , 1971, Nature.

[9]  D. Blackwell,et al.  Ferguson Distributions Via Polya Urn Schemes , 1973 .

[10]  A. F. Smith,et al.  Bayes estimates in one-way and two-way models , 1973 .

[11]  C. Antoniak Mixtures of Dirichlet Processes with Applications to Bayesian Nonparametric Problems , 1974 .

[12]  Kjell Nowak,et al.  A Comment on , 1975 .

[13]  D. Rubin INFERENCE AND MISSING DATA , 1975 .

[14]  V T Farewell,et al.  The analysis of failure times in the presence of competing risks. , 1978, Biometrics.

[15]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[16]  A. Dawid Conditional Independence in Statistical Theory , 1979 .

[17]  D. Berry,et al.  Empirical Bayes Estimation of a Binomial Parameter Via Mixtures of Dirichlet Processes , 1979 .

[18]  D. Basu Randomization Analysis of Experimental Data: The Fisher Randomization Test , 1980 .

[19]  N Gunduz,et al.  Influence of the interval between primary tumor removal and chemotherapy on kinetics and growth of metastases. , 1983, Cancer research.

[20]  D. B. Preston Spectral Analysis and Time Series , 1983 .

[21]  S. Devesa,et al.  Socioeconomic and racial differences in lung cancer incidence. , 1983, American journal of epidemiology.

[22]  T. Ferguson BAYESIAN DENSITY ESTIMATION BY MIXTURES OF NORMAL DISTRIBUTIONS , 1983 .

[23]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  D. Fife,et al.  Incidence, severity, and external causes of pediatric brain injury. , 1986, American journal of diseases of children.

[25]  J. Robins A new approach to causal inference in mortality studies with a sustained exposure period—application to control of the healthy worker survivor effect , 1986 .

[26]  D. Clayton,et al.  Empirical Bayes estimates of age-standardized relative risks for use in disease mapping. , 1987, Biometrics.

[27]  C. Redmond,et al.  Systemic therapy in patients with node-negative breast cancer. A commentary based on two National Surgical Adjuvant Breast and Bowel Project (NSABP) clinical trials. , 1989, Annals of internal medicine.

[28]  C. Redmond,et al.  A randomized clinical trial evaluating tamoxifen in the treatment of patients with node-negative breast cancer who have estrogen-receptor-positive tumors. , 1989, The New England journal of medicine.

[29]  C. Redmond,et al.  A randomized clinical trial evaluating sequential methotrexate and fluorouracil in the treatment of patients with node-negative breast cancer who have estrogen-receptor-negative tumors. , 1989, The New England journal of medicine.

[30]  R. Gray Some diagnostic methods for Cox regression models through hazard smoothing. , 1990, Biometrics.

[31]  N. Gordon,et al.  Application of the theory of finite mixtures for the estimation of 'cure' rates of treated cancer patients. , 1990, Statistics in medicine.

[32]  J. Besag,et al.  Bayesian image restoration, with two applications in spatial statistics , 1991 .

[33]  R. M. Dudley,et al.  Influence Diagrams, Belief Nets and Decision Analysis. , 1991 .

[34]  A. Zauber,et al.  Determinants of late stage diagnosis of breast and cervical cancer: the impact of age, race, social class, and hospital type. , 1991, American journal of public health.

[35]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[36]  Robert Gray,et al.  Flexible Methods for Analyzing Survival Data Using Splines, with Applications to Breast Cancer Prognosis , 1992 .

[37]  Roderick J. A. Little Regression with Missing X's: A Review , 1992 .

[38]  Radford M. Neal Bayesian Mixture Modeling , 1992 .

[39]  Robert Haining,et al.  Statistics for spatial data: by Noel Cressie, 1991, John Wiley & Sons, New York, 900 p., ISBN 0-471-84336-9, US $89.95 , 1993 .

[40]  S. Chib,et al.  Bayesian analysis of binary and polychotomous response data , 1993 .

[41]  J. Nedelman,et al.  Race, socioeconomic status, and the development of end-stage renal disease. , 1994, American journal of kidney diseases : the official journal of the National Kidney Foundation.

[42]  L. Tierney Markov Chains for Exploring Posterior Distributions , 1994 .

[43]  S. MacEachern Estimating normal means with a conjugate style dirichlet process prior , 1994 .

[44]  A. Gelfand,et al.  Efficient parametrisations for normal linear mixed models , 1995 .

[45]  J. Pearl Causal diagrams for empirical research , 1995 .

[46]  K. Chaloner,et al.  Bayesian Experimental Design: A Review , 1995 .

[47]  P. Green Reversible jump Markov chain Monte Carlo computation and Bayesian model determination , 1995 .

[48]  Walter R. Gilks,et al.  Adaptive rejection metropolis sampling , 1995 .

[49]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[50]  C. Redmond,et al.  Sequential methotrexate and fluorouracil for the treatment of node-negative breast cancer patients with estrogen receptor-negative tumors: eight-year results from National Surgical Adjuvant Breast and Bowel Project (NSABP) B-13 and first report of findings from NSABP B-19 comparing methotrexate and , 1996, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[51]  R. Gray Hazard Rate Regression Using Ordinary Nonparametric Regression Smoothers , 1996 .

[52]  Peter Green,et al.  Markov chain Monte Carlo in Practice , 1996 .

[53]  R. Gray,et al.  Annual hazard rates of recurrence for breast cancer after primary therapy. , 1996, Journal of clinical oncology : official journal of the American Society of Clinical Oncology.

[54]  P. Eggers,et al.  Effects of race and income on mortality and use of services among Medicare beneficiaries. , 1996, The New England journal of medicine.

[55]  G. Casella,et al.  Rao-Blackwellisation of sampling schemes , 1996 .

[56]  B Fisher,et al.  Five versus more than five years of tamoxifen therapy for breast cancer patients with negative lymph nodes and estrogen receptor-positive tumors. , 1996, Journal of the National Cancer Institute.

[57]  Radford M. Neal Markov Chain Monte Carlo Methods Based on `Slicing' the Density Function , 1997 .

[58]  Bani K. Mallick,et al.  Hierarchical Generalized Linear Models and Frailty Models with Bayesian Nonparametric Mixing , 1997 .

[59]  Bradley P. Carlin,et al.  BAYES AND EMPIRICAL BAYES METHODS FOR DATA ANALYSIS , 1996, Stat. Comput..

[60]  Michael A. West,et al.  Bayesian forecasting and dynamic models (2nd ed.) , 1997 .

[61]  A. Rukhin Bayes and Empirical Bayes Methods for Data Analysis , 1997 .

[62]  S. MacEachern,et al.  Estimating mixture of dirichlet process models , 1998 .

[63]  Steven N. MacEachern,et al.  Computational Methods for Mixture of Dirichlet Process Models , 1998 .

[64]  H. Chipman,et al.  Bayesian CART Model Search , 1998 .

[65]  J Bound,et al.  Use of census-based aggregate variables to proxy for socioeconomic group: evidence from national samples. , 1998, American journal of epidemiology.

[66]  Adrian F. M. Smith,et al.  A Bayesian CART algorithm , 1998 .

[67]  T. Shakespeare,et al.  Observational Studies , 2003 .

[68]  M E Halloran,et al.  Optimal vaccine trial design when estimating vaccine efficacy for susceptibility and infectiousness from multiple populations. , 1998, Statistics in medicine.

[69]  Bradley P. Carlin,et al.  Markov Chain Monte Carlo in Practice: A Roundtable Discussion , 1998 .

[70]  N M Laird,et al.  Maximum likelihood analysis of generalized linear models with missing covariates , 1999, Statistical methods in medical research.

[71]  M. Daniels A prior for the variance in hierarchical models , 1999 .

[72]  Jun S. Liu,et al.  Sequential importance sampling for nonparametric Bayes models: The next generation , 1999 .

[73]  E. Kolaczyk Bayesian Multiscale Models for Poisson Processes , 1999 .

[74]  P. Damlen,et al.  Gibbs sampling for Bayesian non‐conjugate and hierarchical models by using auxiliary variables , 1999 .

[75]  L Tierney,et al.  Some adaptive monte carlo methods for Bayesian inference. , 1999, Statistics in medicine.

[76]  D J Ferguson,et al.  Dormancy of mammary carcinoma after mastectomy. , 1999, Journal of the National Cancer Institute.

[77]  P. Speckman,et al.  Posterior distribution of hierarchical models using CAR(1) distributions , 1999 .

[78]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[79]  J. Besag,et al.  Bayesian analysis of agricultural field experiments , 1999 .

[80]  J. Hodges,et al.  Effects of processing parameters on physical properties of the silicone maxillofacial prosthetic materials. , 1999, Dental materials : official publication of the Academy of Dental Materials.

[81]  Bradley P. Carlin,et al.  Structured Markov Chain Monte Carlo , 2000 .

[82]  Jim Albert,et al.  Ordinal Data Modeling , 2000 .

[83]  R. Nowak,et al.  A statistical multiscale framework for Poisson inverse problems , 2000, IEEE Trans. Inf. Theory.

[84]  Z He,et al.  Hierarchical Bayes Estimation of Hunting Success Rates with Spatial Correlations , 2000, Biometrics.

[85]  Lingyu Chen,et al.  Exploring Hybrid Monte Carlo in Bayesian Computation , 2000 .

[86]  Radford M. Neal Markov Chain Sampling Methods for Dirichlet Process Mixture Models , 2000 .

[87]  A. Dawid Causal Inference without Counterfactuals , 2000 .

[88]  Xiao-Li Meng,et al.  Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage , 2000 .

[89]  R. J. Hayes,et al.  Design and analysis issues in cluster-randomized trials of interventions against infectious diseases , 2000, Statistical methods in medical research.

[90]  M. Stephens Dealing with label switching in mixture models , 2000 .

[91]  J. Friedman Greedy function approximation: A gradient boosting machine. , 2001 .

[92]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[93]  Dean P. Foster,et al.  Variable Selection in Data Mining: Building a Predictive Model for Bankruptcy , 2001 .

[94]  R Z Omar,et al.  Bayesian methods of analysis for cluster randomized trials with binary outcome data. , 2001, Statistics in medicine.

[95]  Herbert Lee,et al.  Bagging and the Bayesian Bootstrap , 2001, AISTATS.

[96]  J. Hodges,et al.  Counting degrees of freedom in hierarchical and other richly-parameterised models , 2001 .

[97]  Steffen L. Lauritzen,et al.  Causal Inference from Graphical Models , 2001 .

[98]  R. Demicheli,et al.  Does surgery modify growth kinetics of breast cancer micrometastases? , 2001, British Journal of Cancer.

[99]  P. Green,et al.  Modelling Heterogeneity With and Without the Dirichlet Process , 2001 .

[100]  D J Spiegelhalter,et al.  Bayesian methods for cluster randomized trials with continuous responses. , 2001, Statistics in medicine.

[101]  O. Aalen,et al.  Understanding the shape of the hazard rate: A proce ss point of view , 2002 .

[102]  A. Dawid Influence Diagrams for Causal Modelling and Inference , 2002 .

[103]  Bradley P. Carlin,et al.  Bayesian measures of model complexity and fit , 2002 .

[104]  J. Schafer,et al.  Missing data: our view of the state of the art. , 2002, Psychological methods.

[105]  Tom Burr,et al.  Causation, Prediction, and Search , 2003, Technometrics.

[106]  J. Pearl Statistics and causal inference: A review , 2003 .

[107]  Radford M. Neal Slice Sampling , 2003, The Annals of Statistics.

[108]  D. B. Dahl An improved merge-split sampler for conjugate dirichlet process mixture models , 2003 .

[109]  B. Carlin,et al.  On the Precision of the Conditionally Autoregressive Prior in Spatial Models , 2003, Biometrics.

[110]  J. Hodges,et al.  Posterior bimodality in the balanced one‐way random‐effects model , 2003 .

[111]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[112]  David Madigan,et al.  A Sequential Monte Carlo Method for Bayesian Analysis of Massive Datasets , 2003, Data Mining and Knowledge Discovery.

[113]  Michael I. Jordan,et al.  Variational methods for the Dirichlet process , 2004, ICML.

[114]  Jim Q. Smith,et al.  Causal Identification in Design Networks , 2004, MICAI.

[115]  J. Bryant,et al.  S-phase fraction combined with other patient and tumor characteristics for the prognosis of node-negative, estrogen- receptor-positive breast cancer , 2004, Breast Cancer Research and Treatment.

[116]  A. Gelman Parameterization and Bayesian Modeling , 2004 .

[117]  Radford M. Neal,et al.  A Split-Merge Markov chain Monte Carlo Procedure for the Dirichlet Process Mixture Model , 2004 .

[118]  K. Hess,et al.  Estrogen Receptors and Distinct Patterns of Breast Cancer Relapse , 2003, Breast Cancer Research and Treatment.

[119]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[120]  Dongchu Sun,et al.  Hierarchical Bayes estimation of hunting success rates , 1998, Environmental and Ecological Statistics.

[121]  P. Müller,et al.  A Bayesian mixture model for differential gene expression , 2005 .

[122]  Rosalba Miceli,et al.  Time distribution of the recurrence risk for breast cancer patients undergoing mastectomy: Further support about the concept of tumor dormancy , 2005, Breast Cancer Research and Treatment.

[123]  S. Lipsitz,et al.  Missing-Data Methods for Generalized Linear Models , 2005 .

[124]  Charles Hadlock Causality: Models, Reasoning, and Inference , 2005 .

[125]  R. Demicheli,et al.  Does surgery unfavourably perturb the "natural history" of early breast cancer by accelerating the appearance of distant metastases? , 2005, European journal of cancer.

[126]  Xiao-Li Meng,et al.  A BAYESIAN MULTIRESOLUTION HAZARD MODEL WITH APPLICATION TO AN AIDS REPORTING DELAY STUDY , 2005 .

[127]  Ronald A. Howard,et al.  Influence Diagrams , 2005, Decis. Anal..

[128]  A. Gelman Analysis of variance: Why it is more important than ever? , 2005, math/0504499.

[129]  P. Gustafson,et al.  Conservative prior distributions for variance parameters in hierarchical models , 2006 .

[130]  John P Huelsenbeck,et al.  A Dirichlet process model for detecting positive selection in protein-coding DNA sequences. , 2006, Proceedings of the National Academy of Sciences of the United States of America.

[131]  Michael A. West,et al.  Hierarchical priors and mixture models, with applications in regression and density estimation , 2006 .

[132]  M. Wand,et al.  General design Bayesian generalized linear mixed models , 2006, math/0606491.

[133]  Bradley P. Carlin,et al.  Smoothing Balanced Single-Error-Term Analysis of Variance , 2007, Technometrics.

[134]  Roded Sharan,et al.  Bayesian Haplotype Inference via the Dirichlet Process , 2007, J. Comput. Biol..

[135]  Andrew Gelman,et al.  Estimating Incumbency Advantage and Its Variation, as an Example of a Before–After Study , 2008 .

[136]  S. E. Ahmed,et al.  Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference , 2008, Technometrics.

[137]  D.,et al.  Regression Models and Life-Tables , 2022 .