Challenges in Deploying Machine Learning: A Survey of Case Studies

In recent years, machine learning has received increased interest both as an academic research field and as a solution for real-world business problems. However, the deployment of machine learning models in production systems can present a number of issues and concerns. This survey reviews published reports of deploying machine learning solutions in a variety of use cases, industries and applications and extracts practical considerations corresponding to stages of the machine learning deployment workflow. Our survey shows that practitioners face challenges at each stage of the deployment. The goal of this paper is to layout a research agenda to explore approaches addressing these challenges.

[1]  Radu Calinescu,et al.  Assuring the Machine Learning Lifecycle , 2019, ACM Comput. Surv..

[2]  Neil D. Lawrence,et al.  Data Readiness Levels , 2017, ArXiv.

[3]  Tom Diethe,et al.  Continual Learning in Practice , 2019, NeurIPS 2019.

[4]  Eric Tzeng,et al.  Learning a Unified Embedding for Visual Search at Pinterest , 2019, KDD.

[5]  Michael Veale,et al.  Algorithms that remember: model inversion attacks and data protection law , 2018, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[6]  S. Aljunid,et al.  Health-Care Data Collecting, Sharing, and Using in Thailand, China Mainland, South Korea, Taiwan, Japan, and Malaysia , 2012, Value in Health.

[7]  Fabio Roli,et al.  Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning , 2018, CCS.

[8]  Gabriel Dulac-Arnold,et al.  Challenges of Real-World Reinforcement Learning , 2019, ArXiv.

[9]  Barbara Pierscionek,et al.  The Effect of the General Data Protection Regulation on Medical Research , 2017, Journal of medical Internet research.

[10]  Duen Horng Chau,et al.  Firebird: Predicting Fire Risk and Prioritizing Fire Inspections in Atlanta , 2016, KDD.

[11]  M. Mamzer,et al.  Perceptions of artificial intelligence in healthcare: findings from a qualitative survey study among actors in France , 2020, Journal of Translational Medicine.

[12]  D. Sculley,et al.  Hidden Technical Debt in Machine Learning Systems , 2015, NIPS.

[13]  Richard Bowden,et al.  A Survey of Deep Learning Applications to Autonomous Vehicle Control , 2019, IEEE Transactions on Intelligent Transportation Systems.

[14]  Yisroel Mirsky,et al.  The Creation and Detection of Deepfakes , 2020, ACM Comput. Surv..

[15]  Neoklis Polyzotis,et al.  Continuous Training for Production ML in the TensorFlow Extended (TFX) Platform , 2019, OpML.

[16]  Luca Oneto,et al.  Fairness in Machine Learning , 2020, INNSBDDL.

[17]  Ameet Talwalkar,et al.  Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization , 2016, J. Mach. Learn. Res..

[18]  Wan Mohd Nazmee Wan Zainon,et al.  A Review on Data Cleansing Methods for Big Data , 2019, Procedia Computer Science.

[19]  Clive Cox,et al.  Monitoring and explainability of models in production , 2020, ArXiv.

[20]  Fan Zhang,et al.  Stealing Machine Learning Models via Prediction APIs , 2016, USENIX Security Symposium.

[21]  Kendra Albert,et al.  Failure Modes in Machine Learning Systems , 2019, ArXiv.

[22]  Stefan Seebacher,et al.  Challenges in the Deployment and Operation of Machine Learning in Practice , 2019, ECIS.

[23]  Neoklis Polyzotis,et al.  Data Lifecycle Challenges in Production Machine Learning , 2018, SIGMOD Rec..

[24]  Gary Doran,et al.  Enabling Onboard Detection of Events of Scientific Interest for the Europa Clipper Spacecraft , 2019, KDD.

[25]  J. C. Schlimmer,et al.  Incremental learning from noisy data , 2004, Machine Learning.

[26]  Tim W. Nattkemper,et al.  Gear-Induced Concept Drift in Marine Images and Its Effect on Deep Learning Classification , 2020, Frontiers in Marine Science.

[27]  Andrew Nguyen,et al.  ARTIFICIAL INTELLIGENCE FOR THE REAL WORLD , 2023, International Research Journal of Modernization in Engineering Technology and Science.

[28]  Magnus Nyström,et al.  Adversarial Machine Learning - Industry Perspectives , 2020, SSRN Electronic Journal.

[29]  Abbas Keramati,et al.  Developing a prediction model for customer churn from electronic banking services using data mining , 2016, Financial Innovation.

[30]  Isaac L. Chuang,et al.  Confident Learning: Estimating Uncertainty in Dataset Labels , 2019, J. Artif. Intell. Res..

[31]  Chang-Tien Lu,et al.  EMBERS at 4 years: Experiences operating an Open Source Indicators Forecasting System , 2016, KDD.

[32]  Sameer Singh,et al.  Beyond Accuracy: Behavioral Testing of NLP Models with CheckList , 2020, ACL.

[33]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[34]  Andrew McCallum,et al.  Energy and Policy Considerations for Deep Learning in NLP , 2019, ACL.

[35]  Pascual Campoy Cervera,et al.  A Review of Deep Learning Methods and Applications for Unmanned Aerial Vehicles , 2017, J. Sensors.

[36]  Somesh Jha,et al.  Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures , 2015, CCS.

[37]  Ramprakash Ramamoorthy,et al.  {AI} from Labs to Production - Challenges and Learnings , 2019 .

[38]  Andrés R. Masegosa,et al.  Analyzing concept drift: A case study in the financial sector , 2020, Intell. Data Anal..

[39]  Neoklis Polyzotis,et al.  Data Validation for Machine Learning , 2019, MLSys.

[40]  Tao Xu,et al.  Applying Deep Learning to Airbnb Search , 2018, KDD.

[41]  David Bull,et al.  Artificial intelligence in the creative industries: a review , 2020, Artif. Intell. Rev..

[42]  Alexander Lavin,et al.  Technology Readiness Levels for Machine Learning Systems , 2020, ArXiv.

[43]  Li Yang,et al.  On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice , 2020, Neurocomputing.

[44]  Pablo Estevez,et al.  150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com , 2019, KDD.

[45]  Tony Doyle,et al.  Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy , 2017, Inf. Soc..

[46]  Ursula Challita,et al.  When Machine Learning Meets Wireless Cellular Networks: Deployment, Challenges, and Applications , 2020, IEEE Communications Magazine.

[47]  Christopher Ré,et al.  Snuba: Automating Weak Supervision to Label Training Data , 2018, Proc. VLDB Endow..

[48]  Manzoor Ahmed Hashmani,et al.  A Critical Review on Adverse Effects of Concept Drift over Machine Learning Classification Models , 2020 .

[49]  AzureML Team,et al.  AzureML: Anatomy of a machine learning service , 2016, PAPIs.

[50]  Bernhard Kainz,et al.  A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis , 2019, Medical Image Anal..

[51]  Kumar Venkateswar.  Using Amazon SageMaker to Operationalize Machine Learning , 2019 .

[52]  Robert Soden,et al.  Taking Ethics, Fairness, and Bias Seriously in Machine Learning for Disaster Risk Management , 2019, ArXiv.

[53]  Peng Huang,et al.  AIOps: Real-World Challenges and Research Innovations , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion).

[54]  Yoav Shoham,et al.  The Cost of Training NLP Models: A Concise Overview , 2020, ArXiv.

[55]  Cedric Baudoin,et al.  Towards the Deployment of Machine Learning Solutions in Network Traffic Classification: A Systematic Survey , 2019, IEEE Communications Surveys & Tutorials.

[56]  Michael Stonebraker,et al.  Detecting Data Errors: Where are we and what needs to be done? , 2016, Proc. VLDB Endow..

[58]  Ankur Taly,et al.  Explainable machine learning in deployment , 2019, FAT*.

[59]  Chang Liu,et al.  Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning , 2018, 2018 IEEE Symposium on Security and Privacy (SP).

[60]  Harald C. Gall,et al.  Software Engineering for Machine Learning: A Case Study , 2019, 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP).

[61]  Joe Walsh,et al.  Deploying Machine Learning Models for Public Policy: A Framework , 2018, KDD.

[62]  David Leslie,et al.  Understanding artificial intelligence ethics and safety , 2019, ArXiv.

[63]  Siril Yella,et al.  Machine Learning Algorithms in Heavy Process Manufacturing , 2016 .

[64]  Joaquin Vanschoren,et al.  Adaptation Strategies for Automated Machine Learning on Evolving Data , 2020, ArXiv.

[65]  Michael Gao,et al.  "The human body is a black box": supporting clinical decision-making with deep learning , 2019, FAT*.

[66]  Miryung Kim,et al.  Data Scientists in Software Teams: State of the Art and Challenges , 2018, IEEE Transactions on Software Engineering.

[67]  Diana Marculescu,et al.  Hardware-Aware Machine Learning: Modeling and Optimization , 2018, 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD).

[68]  Jeffrey Heer,et al.  Enterprise Data Analysis and Visualization: An Interview Study , 2012, IEEE Transactions on Visualization and Computer Graphics.

[69]  Samy Bengio,et al.  Adversarial Machine Learning at Scale , 2016, ICLR.

[70]  Michael Affenzeller,et al.  Machine learning based concept drift detection for predictive maintenance , 2019, Comput. Ind. Eng..

[71]  Michal Pechoucek,et al.  Industrial deployment of multi-agent technologies: review and selected case studies , 2008, Autonomous Agents and Multi-Agent Systems.

[72]  Jimmy J. Lin,et al.  Scaling big data mining infrastructure: the twitter experience , 2013, SKDD.

[73]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[74]  Christopher Ré,et al.  Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale , 2018, SIGMOD Conference.

[75]  知秀 柴田.  5分で分かる!? 有名論文ナナメ読み：Jacob Devlin et al. : BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding , 2020 .