Privacy Aware Learning

We study statistical risk minimization problems under a privacy model in which the data is kept confidential even from the learner. In this local privacy framework, we establish sharp upper and lower bounds on the convergence rates of statistical estimation procedures. As a consequence, we exhibit a precise tradeoff between the amount of privacy the data preserves and the utility, as measured by convergence rate, of any statistical estimator or learning procedure.

[1]  A. Wald.  Contributions to the Theory of Statistical Estimation and Testing Hypotheses , 1939 .

[2]  S L Warner,et al.  Randomized response: a survey technique for eliminating evasive answer bias. , 1965, Journal of the American Statistical Association.

[3]  R. Phelps.  Lectures on Choquet's Theorem , 1966 .

[4]  Ivan P. Fellegi,et al.  On the Question of Statistical Confidentiality , 1972 .

[5]  L. Lecam.  Convergence of Estimates Under Dimensionality Restrictions , 1973 .

[6]  O. Mangasarian.  Uniqueness of solution in linear programming , 1979 .

[7]  John Darzentas,et al.  Problem Complexity and Method Efficiency in Optimization , 1983 .

[8]  Patrick Billingsley,et al.  Probability and Measure. , 1986 .

[9]  George T. Duncan,et al.  Disclosure-Limited Data Dissemination , 1986 .

[10]  D. Lambert,et al.  The Risk of Disclosure for Microdata , 1989 .

[11]  John N. Tsitsiklis,et al.  Parallel and distributed computation , 1989 .

[12]  R. Gray.  Entropy and Information Theory , 1990, Springer New York.

[13]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[14]  Boris Polyak,et al.  Acceleration of stochastic approximation by averaging , 1992 .

[15]  J. Hiriart-Urruty,et al.  Convex analysis and minimization algorithms , 1993 .

[16]  Michael Kearns,et al.  Efficient noise-tolerant learning from statistical queries , 1993, STOC.

[17]  Bin Yu.  Assouad, Fano, and Le Cam , 1997 .

[18]  O. Kallenberg.  Foundations of Modern Probability , 2021, Probability Theory and Stochastic Modelling.

[19]  Yuhong Yang,et al.  Information-theoretic determination of minimax rates of convergence , 1999 .

[20]  Alexandre V. Evfimievski,et al.  Limiting privacy breaches in privacy preserving data mining , 2003, PODS.

[21]  Marc Teboulle,et al.  Mirror descent and nonlinear projected subgradient methods for convex optimization , 2003, Oper. Res. Lett..

[22]  Irit Dinur,et al.  Revealing information while preserving privacy , 2003, PODS.

[23]  Martin Zinkevich,et al.  Online Convex Programming and Generalized Infinitesimal Gradient Ascent , 2003, ICML.

[24]  Jerome P. Reiter.  Estimating Risks of Identification Disclosure in Microdata , 2005 .

[25]  Thomas M. Cover,et al.  Elements of Information Theory: Cover/Elements of Information Theory, Second Edition , 2005 .

[26]  Anna Oganian,et al.  A Framework for Evaluating the Utility of Data Altered to Protect Confidentiality , 2006 .

[27]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[28]  Thomas M. Cover,et al.  Elements of information theory (2. ed.) , 2006 .

[29]  Dan Suciu,et al.  Journal of the ACM , 2006 .

[30]  Cynthia Dwork,et al.  Calibrating Noise to Sensitivity in Private Data Analysis , 2006, TCC.

[31]  Larry A. Wasserman,et al.  Compressed Regression , 2007, NIPS.

[32]  L. Wasserman,et al.  A Statistical Framework for Differential Privacy , 2008, 0811.2501.

[33]  Adam D. Smith,et al.  Composition attacks and auxiliary information in data privacy , 2008, KDD.

[34]  A. Blum,et al.  A learning theory approach to non-interactive database privacy , 2008, STOC.

[35]  Cynthia Dwork,et al.  Differential Privacy: A Survey of Results , 2008, TAMC.

[36]  Sofya Raskhodnikova,et al.  What Can We Learn Privately? , 2008, 2008 49th Annual IEEE Symposium on Foundations of Computer Science.

[37]  Martin J. Wainwright,et al.  Information-theoretic lower bounds on the oracle complexity of convex optimization , 2009, NIPS.

[38]  Tim Roughgarden,et al.  Universally utility-maximizing privacy mechanisms , 2008, STOC '09.

[39]  Alexander Shapiro,et al.  Stochastic Approximation approach to Stochastic Programming , 2013 .

[40]  Larry A. Wasserman,et al.  Differential privacy with compression , 2009, 2009 IEEE International Symposium on Information Theory.

[41]  Cynthia Dwork,et al.  Differential privacy and robust statistics , 2009, STOC '09.

[42]  Shlomo Shamai,et al.  Information Theoretic Security , 2009, Found. Trends Commun. Inf. Theory.

[43]  Larry A. Wasserman,et al.  Compressed and Privacy-Sensitive Sparse Regression , 2009, IEEE Transactions on Information Theory.

[44]  H. Vincent Poor,et al.  An information-theoretic approach to privacy , 2010, 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[45]  Kunal Talwar,et al.  On the geometry of differential privacy , 2009, STOC '10.

[46]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[47]  Guy N. Rothblum,et al.  Boosting and Differential Privacy , 2010, 2010 IEEE 51st Annual Symposium on Foundations of Computer Science.

[48]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[49]  Alan F. Karr,et al.  Risk‐Utility Paradigms for Statistical Disclosure Limitation: How to Think, But Not How to Act , 2011 .

[50]  Adam D. Smith,et al.  Privacy-preserving statistical estimation with optimal convergence rates , 2011, STOC '11.

[51]  Anand D. Sarwate,et al.  Differentially Private Empirical Risk Minimization , 2009, J. Mach. Learn. Res..

[52]  Ling Huang,et al.  Learning in a Large Function Space: Privacy-Preserving Mechanisms for SVM Learning , 2009, J. Priv. Confidentiality.

[53]  Martin J. Wainwright,et al.  Information-Theoretic Lower Bounds on the Oracle Complexity of Stochastic Convex Optimization , 2010, IEEE Transactions on Information Theory.

[54]  Michael I. Jordan,et al.  Local Privacy and Statistical Minimax Rates , 2013, 2013 IEEE 54th Annual Symposium on Foundations of Computer Science.

[55]  J. Norris.  Appendix: probability and measure , 1997 .

[56]  Larry A. Wasserman,et al.  Random Differential Privacy , 2011, J. Priv. Confidentiality.

[57]  Martin J. Wainwright,et al.  Local Privacy, Data Processing Inequalities, and Statistical Minimax Rates , 2013, 1302.3203.

[58]  Adam D. Smith,et al.  The Power of Linear Reconstruction Attacks , 2012, SODA.

[59]  Aleksandar Nikolov,et al.  The geometry of differential privacy: the sparse and approximate cases , 2012, STOC '13.

[60]  Carles Padró,et al.  Information Theoretic Security , 2013, Lecture Notes in Computer Science.

[61]  Martin J. Wainwright,et al.  Privacy Aware Learning , 2012, JACM.