A Survey of Deep Active Learning

Active learning (AL) attempts to maximize a model’s performance gain while annotating the fewest samples possible. Deep learning (DL) is greedy for data and requires a large amount of data supply to optimize a massive number of parameters if the model is to learn how to extract high-quality features. In recent years, due to the rapid development of internet technology, we have entered an era of information abundance characterized by massive amounts of available data. As a result, DL has attracted significant attention from researchers and has been rapidly developed. Compared with DL, however, researchers have a relatively low interest in AL. This is mainly because before the rise of DL, traditional machine learning requires relatively few labeled samples, meaning that early AL is rarely according the value it deserves. Although DL has made breakthroughs in various fields, most of this success is due to a large number of publicly available annotated datasets. However, the acquisition of a large number of high-quality annotated datasets consumes a lot of manpower, making it unfeasible in fields that require high levels of expertise (such as speech recognition, information extraction, medical images, etc.). Therefore, AL is gradually coming to receive the attention it is due. It is therefore natural to investigate whether AL can be used to reduce the cost of sample annotation while retaining the powerful learning capabilities of DL. As a result of such investigations, deep active learning (DeepAL) has emerged. Although research on this topic is quite abundant, there has not yet been a comprehensive survey of DeepAL-related works; accordingly, this article aims to fill this gap. We provide a formal classification method for the existing work, along with a comprehensive and systematic overview. In addition, we also analyze and summarize the development of DeepAL from an application perspective. Finally, we discuss the confusion and problems associated with DeepAL and provide some possible development directions.

[1]  Devis Tuia,et al.  Half a Percent of Labels is Enough: Efficient Animal Detection in UAV Imagery Using Deep CNNs and Active Learning , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[2]  Mikkel Baun Kjærgaard,et al.  Smart Devices are Different: Assessing and MitigatingMobile Sensing Heterogeneities for Activity Recognition , 2015, SenSys.

[3]  Bernt Schiele,et al.  CityPersons: A Diverse Dataset for Pedestrian Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  William Yang Wang.  “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection , 2017, ACL.

[5]  Sotaro Tsukizawa,et al.  Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Diane J Cook,et al.  Assessing the Quality of Activities in a Smart Environment , 2009, Methods of Information in Medicine.

[7]  Bogdan Kwolek,et al.  Breast Cancer Classification on Histopathological Images Affected by Data Imbalance Using Active Learning and Deep Convolutional Neural Network , 2019, ICANN.

[8]  Rong Jin,et al.  Batch mode active learning and its application to medical image classification , 2006, ICML.

[9]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[10]  Nikolaos Papanikolopoulos,et al.  Multi-class batch-mode active learning for image classification , 2010, 2010 IEEE International Conference on Robotics and Automation.

[11]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[12]  Jie Yang,et al.  Leveraging Crowdsourcing Data for Deep Active Learning An Application: Learning Intents in Alexa , 2018, WWW.

[13]  Franziska Abend,et al.  Facility Location Concepts Models Algorithms And Case Studies , 2016 .

[14]  Klaus Brinker,et al.  Incorporating Diversity in Active Learning with Support Vector Machines , 2003, ICML.

[15]  Xiaojun Chang,et al.  A Comprehensive Survey of Neural Architecture Search , 2020, ACM Comput. Surv..

[16]  Nirmalya Roy,et al.  SenseBox: A low-cost smart home system , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[17]  Remus Pop,et al.  Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles , 2018, ArXiv.

[18]  Erik F. Tjong Kim Sang,et al.  Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition , 2003, CoNLL.

[19]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[20]  Frédéric Precioso,et al.  Adversarial Active Learning for Deep Networks: a Margin Based Approach , 2018, ArXiv.

[21]  Bo Pang,et al.  Thumbs up? Sentiment Classification using Machine Learning Techniques , 2002, EMNLP.

[22]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[23]  Hanno Gottschalk,et al.  Deep Bayesian Active Semi-Supervised Learning , 2018, ICMLA.

[24]  Xiaotong Shen,et al.  high-dimensional data analysis , 1991 .

[25]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[26]  Horst Bischof,et al.  Person Re-identification by Descriptive and Discriminative Classification , 2011, SCIA.

[27]  Xiaolong Wang,et al.  Active deep learning method for semi-supervised sentiment classification , 2013, Neurocomputing.

[28]  Andrew McCallum,et al.  Overcoming Practical Issues of Deep Active Learning and its Applications on Named Entity Recognition , 2019, ArXiv.

[29]  Stefan Wrobel,et al.  Active Hidden Markov Models for Information Extraction , 2001, IDA.

[30]  Di He,et al.  Multilingual Neural Machine Translation with Knowledge Distillation , 2019, ICLR.

[31]  Jeffrey M. Hausdorff,et al.  Wearable Assistant for Parkinson’s Disease Patients With the Freezing of Gait Symptom , 2010, IEEE Transactions on Information Technology in Biomedicine.

[32]  Dan Wang,et al.  A new active labeling method for deep learning , 2014, 2014 International Joint Conference on Neural Networks (IJCNN).

[33]  Yue Huang,et al.  Cost-Effective Vehicle Type Recognition in Surveillance Images With Deep Active Learning and Web Data , 2020, IEEE Transactions on Intelligent Transportation Systems.

[34]  Liu Yang,et al.  Batch Mode Active Learning for Semantic Segmentation Based on Multi-Clue Sample Selection , 2019, CIKM.

[35]  Martti Vainio,et al.  Proceedings of the Annual Conference of the International Speech Communication Association , 2016, Interspeech 2016.

[36]  Andreas Niekler,et al.  A Survey of Active Learning for Text Classification using Deep Neural Networks , 2020, ArXiv.

[37]  Leah S. Larkey,et al.  A patent search and classification system , 1999, DL '99.

[38]  C. Lintott,et al.  Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna , 2015, Scientific Data.

[39]  Jörg Tiedemann,et al.  Parallel Data, Tools and Interfaces in OPUS , 2012, LREC.

[40]  Mohammad Sadegh Norouzzadeh,et al.  A deep active learning system for species identification and counting in camera trap images , 2019, Methods in Ecology and Evolution.

[41]  Zhi Zhang,et al.  Bag of Tricks for Image Classification with Convolutional Neural Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[42]  Khaled Shaalan,et al.  Speech Recognition Using Deep Neural Networks: A Systematic Review , 2019, IEEE Access.

[43]  DeActive: Scaling Activity Recognition with Active Deep Learning , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[44]  Adrian E. Roitberg,et al.  Less is more: sampling chemical space with active learning , 2018, The Journal of chemical physics.

[45]  Junhao Wen,et al.  Fundus Image Classification Using VGG-19 Architecture with PCA and SVD , 2018, Symmetry.

[46]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  K. Vijay-Shanker,et al.  Taking into Account the Differences between Actively and Passively Acquired Data: The Case of Active Learning with Support Vector Machines for Imbalanced Datasets , 2009, NAACL.

[48]  A. McCallum,et al.  Using error decay prediction to overcome practical issues of deep active learning for named entity recognition , 2019, Machine Learning.

[49]  Ashish Kapoor,et al.  Active learning for large multi-class problems , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  M. Cha,et al.  Rumor Detection over Varying Time Windows , 2017, PloS one.

[51]  Giovanna Martínez-Arellano,et al.  Towards An Active Learning Approach To Tool Condition Monitoring With Bayesian Deep Learning , 2019, ECMS.

[52]  Roberto Cipolla,et al.  Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding , 2015, BMVC.

[53]  Bang An,et al.  Deep Active Learning for Text Classification , 2018, ICVISP.

[54]  Hae Young Noh,et al.  O‐MedAL: Online active deep learning for medical image analysis , 2019, WIREs Data Mining Knowl. Discov..

[55]  Zoubin Ghahramani,et al.  Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference , 2015, ArXiv.

[56]  Pietro Perona,et al.  Pedestrian Detection: An Evaluation of the State of the Art , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[57]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[58]  Andrew McCallum,et al.  Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking , 2018, ACL.

[59]  Yoshua Bengio,et al.  Greedy Layer-Wise Training of Deep Networks , 2006, NIPS.

[60]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[61]  Colin Raffel,et al.  Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition , 2019, ICML.

[62]  José Miguel Hernández-Lobato,et al.  Bayesian Batch Active Learning as Sparse Subset Approximation , 2019, NeurIPS.

[63]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[64]  Bharadwaj Veeravalli,et al.  Deeply Supervised Active Learning for Finger Bones Segmentation , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[65]  Ronald Rosenfeld,et al.  Semi-supervised learning with graphs , 2005 .

[66]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[67]  Philippe Cudré-Mauroux,et al.  ActiveLink: Deep Active Learning for Link Prediction in Knowledge Graphs , 2019, WWW.

[68]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[69]  Ruiwei Feng,et al.  Nodule-Plus R-CNN and Deep Self-Paced Active Learning for 3D Instance Segmentation of Pulmonary Nodules , 2019, IEEE Access.

[70]  B. S. Manjunath,et al.  CellECT: cell evolution capturing tool , 2016, BMC Bioinformatics.

[71]  Jose M. Alvarez,et al.  The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning , 2018, ArXiv.

[72]  Ran El-Yaniv,et al.  Deep Active Learning over the Long Tail , 2017, ArXiv.

[73]  Min Chen,et al.  Deep Learning for Imbalanced Multimedia Data Classification , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[74]  Maneesh Singh,et al.  Sampling Bias in Deep Active Classification: An Empirical Study , 2019, EMNLP.

[75]  K. Bretonnel Cohen,et al.  A shared task involving multi-label classification of clinical free text , 2007, BioNLP@ACL.

[76]  Scott Doyle,et al.  Active deep learning: Improved training efficiency of convolutional neural networks for tissue classification in oral cavity cancer , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[77]  Ye Zhang,et al.  Active Discriminative Text Representation Learning , 2016, AAAI.

[78]  Mark Dredze,et al.  A large-scale quantitative analysis of latent factors and sentiment in online doctor reviews , 2014, J. Am. Medical Informatics Assoc..

[79]  Venkatesh Umaashankar,et al.  ActiveHARNet: Towards On-Device Deep Bayesian Active Learning for Human Activity Recognition , 2019, EMDL '19.

[80]  Angela P. Schoellig,et al.  Active Training Trajectory Generation for Inverse Dynamics Model Learning with Deep Neural Networks , 2019, 2019 IEEE 58th Conference on Decision and Control (CDC).

[81]  Chao Li,et al.  Active Transfer Learning Network: A Unified Deep Joint Spectral–Spatial Feature Learning Model for Hyperspectral Image Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[82]  Serena Yeung,et al.  Adversarial Representation Active Learning , 2019, ArXiv.

[83]  Min Chen,et al.  Efficient Imbalanced Multimedia Concept Retrieval by Deep Learning on Spark Clusters , 2017, Int. J. Multim. Data Eng. Manag..

[84]  Kwang In Kim,et al.  Task-Aware Variational Adversarial Active Learning , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[85]  Gustavo Carneiro,et al.  Bayesian Generative Active Deep Learning , 2019, ICML.

[86]  Kyunghyun Paeng,et al.  A Robust and Effective Approach Towards Accurate Metastasis Detection and pN-stage Classification in Breast Cancer , 2018, MICCAI.

[87]  Melih Kandemir,et al.  Deep Active Learning with Adaptive Acquisition , 2019, IJCAI.

[88]  Nikolaos Papanikolopoulos,et al.  Multi-class active learning for image classification , 2009, CVPR.

[89]  Mohamed A. Ismail,et al.  Multi-level gene/MiRNA feature selection using deep belief nets and active learning , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[90]  Katherine J. Kuchenbecker,et al.  How Much Haptic Surface Data Is Enough? , 2017, AAAI Spring Symposia.

[91]  Trevor Darrell,et al.  BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling , 2018, ArXiv.

[92]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[93]  Peter Robinson,et al.  Mind reading machines: automated inference of cognitive mental states from video , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[94]  Pedro Costa,et al.  MedAL: Accurate and Robust Deep Active Learning for Medical Image Analysis , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[95]  Daphne Koller,et al.  Active learning: theory and applications , 2001 .

[96]  Trevor Campbell,et al.  Coresets for Scalable Bayesian Logistic Regression , 2016, NIPS.

[97]  William A. Gale,et al.  A sequential algorithm for training text classifiers , 1994, SIGIR '94.

[98]  Xiang Zhang,et al.  Character-level Convolutional Networks for Text Classification , 2015, NIPS.

[99]  Xiao Lin,et al.  Active Learning for Visual Question Answering: An Empirical Study , 2017, ArXiv.

[100]  Cynthia Brandt,et al.  Semi-supervised clinical text classification with Laplacian SVMs: An application to cancer case management , 2013, J. Biomed. Informatics.

[101]  Mark Craven,et al.  Multiple-Instance Active Learning , 2007, NIPS.

[102]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[103]  Naif Alajlan,et al.  Deep learning approach for active classification of electrocardiogram signals , 2016, Inf. Sci..

[104]  Jose M. Alvarez,et al.  Large-Scale Visual Active Learning with Deep Probabilistic Ensembles , 2018, ArXiv.

[105]  Qinghua Zheng,et al.  Deep Similarity-Based Batch Mode Active Learning with Exploration-Exploitation , 2017, 2017 IEEE International Conference on Data Mining (ICDM).

[106]  Sophia Ananiadou,et al.  A Text Mining Pipeline Using Active and Deep Learning Aimed at Curating Information in Computational Neuroscience , 2018, Neuroinformatics.

[107]  Shuying Li,et al.  Active-Learning-Incorporated Deep Transfer Learning for Hyperspectral Image Classification , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[108]  G. Griffin,et al.  Caltech-256 Object Category Dataset , 2007 .

[109]  Pavlos Protopapas,et al.  Cost-Sensitive Batch Mode Active Learning: Designing Astronomical Observation by Optimizing Telescope Time and Telescope Choice , 2016, SDM.

[110]  Lin Yang,et al.  Coarse-to-Fine Stacked Fully Convolutional Nets for lymph node segmentation in ultrasound images , 2016, 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[111]  Alexander Panchenko,et al.  Active Learning with Deep Pre-trained Models for Sequence Tagging of Clinical and Biomedical Texts , 2019, 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

[112]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[113]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[114]  Shlomo Argamon,et al.  Committee-Based Sampling For Training Probabilistic Classi(cid:12)ers , 1995 .

[115]  Bernhard Kainz,et al.  A Survey on Active Learning and Human-in-the-Loop Deep Learning for Medical Image Analysis , 2019, Medical Image Anal..

[116]  Nassir Navab,et al.  Multiclass Deep Active Learning for Detecting Red Blood Cell Subtypes in Brightfield Microscopy , 2019, MICCAI.

[117]  Philip S. Yu,et al.  Inferring anchor links across multiple heterogeneous social networks , 2013, CIKM.

[118]  Changjian Shui,et al.  Deep Active Learning: Unified and Principled Method for Query and Training , 2019, AISTATS.

[119]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[120]  Joost van de Weijer,et al.  Active Learning for Deep Detection Neural Networks , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[121]  Gregory Piatetsky-Shapiro,et al.  High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality , 2000 .

[122]  A. Törcsvári,et al.  Automated categorization in the international patent classification , 2003, SIGF.

[123]  Koby Crammer,et al.  A theory of learning from different domains , 2010, Machine Learning.

[124]  Sethuraman Panchanathan,et al.  Deep active learning for image classification , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[125]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[126]  Anima Anandkumar,et al.  Deep Active Learning for Named Entity Recognition , 2017, Rep4NLP@ACL.

[127]  Carlos Busso,et al.  Active Learning for Speech Emotion Recognition Using Deep Neural Network , 2019, 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII).

[128]  Zhiyong Lu,et al.  NCBI disease corpus: A resource for disease name recognition and concept normalization , 2014, J. Biomed. Informatics.

[129]  Sanda M Harabagiu,et al.  Active deep learning for the identification of concepts and relations in electroencephalography reports , 2019, J. Biomed. Informatics.

[130]  Ramesh Raskar,et al.  Designing Neural Network Architectures using Reinforcement Learning , 2016, ICLR.

[131]  Jeff M. Phillips,et al.  Coresets and Sketches , 2016, ArXiv.

[132]  Barnabás Póczos,et al.  Competence-based Curriculum Learning for Neural Machine Translation , 2019, NAACL.

[133]  Bo Du,et al.  A batch-mode active learning framework by querying discriminative and representative samples for hyperspectral image classification , 2016, Neurocomputing.

[134]  Fedor Zhdanov,et al.  Diverse mini-batch Active Learning , 2019, ArXiv.

[135]  Zoubin Ghahramani,et al.  Bayesian Active Learning for Classification and Preference Learning , 2011, ArXiv.

[136]  Volker Eiselein,et al.  Deep Active Learning for In Situ Plankton Classification , 2018, CVAUI/IWCF/MIPPSNA@ICPR.

[137]  Jitendra Malik,et al.  Are All Training Examples Created Equal? An Empirical Study , 2018, ArXiv.

[138]  D. Angluin.  Queries and Concept Learning , 1988 .

[139]  Pratim Ghosh,et al.  A Linear Program Formulation for the Segmentation of Ciona Membrane Volumes , 2013, MICCAI.

[140]  Vincent Ng,et al.  Mine the Easy, Classify the Hard: A Semi-Supervised Approach to Automatic Sentiment Classification , 2009, ACL.

[141]  Vikram Krishnamurthy,et al.  Algorithms for optimal scheduling and management of hidden Markov model sensors , 2002, IEEE Trans. Signal Process..

[142]  Arnold W. M. Smeulders,et al.  Active learning using pre-clustering , 2004, ICML.

[143]  Jun Nakanishi,et al.  Dynamic motion learning for multi-DOF flexible-joint robots using active–passive motor babbling through deep learning , 2017, Adv. Robotics.

[144]  Ryan P. Adams,et al.  Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks , 2015, ICML.

[145]  Marcin Junczys-Dowmunt,et al.  The United Nations Parallel Corpus v1.0 , 2016, LREC.

[146]  Burr Settles,et al.  Active Learning Literature Survey , 2009 .

[147]  Charles A. Sutton,et al.  VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning , 2017, NIPS.

[148]  Richard C. Pais,et al.  The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. , 2011, Medical physics.

[149]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[150]  Silvio Savarese,et al.  A Geometric Approach to Active Learning for Convolutional Neural Networks , 2017, ArXiv.

[151]  Zachary C. Lipton,et al.  Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study , 2018, EMNLP.

[152]  Huikai Shao,et al.  Building an Active Palmprint Recognition System , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[153]  Aude Billard,et al.  On Learning, Representing, and Generalizing a Task in a Humanoid Robot , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[154]  Luiz Eduardo Soares de Oliveira,et al.  A Dataset for Breast Cancer Histopathological Image Classification , 2016, IEEE Transactions on Biomedical Engineering.

[155]  S. G,et al.  Classification of ECG beats using deep belief network and active learning. , 2018, Medical & biological engineering & computing.

[156]  Orhan Firat,et al.  Massively Multilingual Neural Machine Translation , 2019, NAACL.

[157]  Ashit Talukder,et al.  Active learning based news veracity detection with feature weighting and deep-shallow fusion , 2017, 2017 IEEE International Conference on Big Data (Big Data).

[158]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[159]  In So Kweon,et al.  Learning Loss for Active Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[160]  Trevor Darrell,et al.  Fully convolutional networks for semantic segmentation , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[161]  Vinay P. Namboodiri,et al.  Deep active learning for object detection , 2018, BMVC.

[162]  H. Sebastian Seung,et al.  Query by committee , 1992, COLT '92.

[163]  B. S. Manjunath,et al.  Membrane segmentation via active learning with deep networks , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[164]  Ronan Collobert,et al.  wav2vec: Unsupervised Pre-training for Speech Recognition , 2019, INTERSPEECH.

[165]  Margaret Mitchell,et al.  VQA: Visual Question Answering , 2015, International Journal of Computer Vision.

[166]  Jia Wu,et al.  Deep Active Learning for Anchor User Prediction , 2019, IJCAI.

[167]  Christopher H. Bryant,et al.  Functional genomic hypothesis generation and experimentation by a robot scientist , 2004, Nature.

[168]  Elmar Haussmann,et al.  Training Data Distribution Search with Ensemble Active Learning , 2019 .

[169]  Xuhui Chen,et al.  Cost-Sensitive Deep Active Learning for Epileptic Seizure Detection , 2018, BCB.

[170]  Daphne Koller,et al.  Self-Paced Learning for Latent Variable Models , 2010, NIPS.

[171]  Jingbo Zhu,et al.  Learning Deep Transformer Models for Machine Translation , 2019, ACL.

[172]  Yunhui Yan,et al.  A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects , 2013 .

[173]  Eduardo P. Wiechmann,et al.  Active learning for clinical text classification: is it better than random sampling? , 2012, J. Am. Medical Informatics Assoc..

[174]  Yifei Lu,et al.  Deep Active Self-paced Learning for Accurate Pulmonary Nodule Segmentation , 2018, MICCAI.

[175]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[176]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[177]  Daphne Koller,et al.  Support Vector Machine Active Learning with Applications to Text Classification , 2002, J. Mach. Learn. Res..

[178]  Tom M. Mitchell,et al.  Contextual Parameter Generation for Universal Neural Machine Translation , 2018, EMNLP.

[179]  Frederic B. Fitch,et al.  McCulloch Warren S. and Pitts Walter. A logical calculus of the ideas immanent in nervous activity. Bulletin of mathematical biophysics , vol. 5 (1943), pp. 115–133. , 1944, Journal of Symbolic Logic.

[180]  Trevor Campbell,et al.  Automated Scalable Bayesian Inference via Hilbert Coresets , 2017, J. Mach. Learn. Res..

[181]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[182]  Daniel Cremers,et al.  CAPTCHA Recognition with Active Deep Learning , 2015 .

[183]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[184]  Carlos Busso,et al.  Building Naturalistic Emotionally Balanced Speech Corpus by Retrieving Emotional Speech from Existing Podcast Recordings , 2019, IEEE Transactions on Affective Computing.

[185]  Yarin Gal,et al.  BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning , 2019, NeurIPS.

[186]  Zhenshen Qu,et al.  Deep Active Learning for Remote Sensing Object Detection , 2020, ArXiv.

[187]  Yee Whye Teh,et al.  Bayesian Learning via Stochastic Gradient Langevin Dynamics , 2011, ICML.

[188]  Ashish Kapoor,et al.  Active learning for large multi-class problems , 2009, CVPR.

[189]  Michael S. Bernstein,et al.  Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations , 2016, International Journal of Computer Vision.

[190]  Trevor Darrell,et al.  Variational Adversarial Active Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[191]  Quoc V. Le,et al.  SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition , 2019, INTERSPEECH.

[192]  Mohamed Medhat Gaber,et al.  Deep Active Learning for Autonomous Navigation , 2016, EANN.

[193]  Christian S. Perone,et al.  Deep Active Learning for Axon-Myelin Segmentation on Histology Data , 2019, ArXiv.

[194]  Andrew McCallum,et al.  Toward Optimal Active Learning through Monte Carlo Estimation of Error Reduction , 2001, ICML 2001.

[195]  Ruimao Zhang,et al.  Cost-Effective Active Learning for Deep Image Classification , 2017, IEEE Transactions on Circuits and Systems for Video Technology.

[196]  Xiao Fu,et al.  Deep Active Learning for Surface Defect Detection , 2020, Sensors.

[197]  Yu Wu,et al.  Exploit the Unknown Gradually: One-Shot Video-Based Person Re-identification by Stepwise Learning , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[198]  Trevor Darrell,et al.  BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling , 2018, ArXiv.

[199]  Shaogang Gong,et al.  Deep Reinforcement Active Learning for Human-in-the-Loop Person Re-Identification , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[200]  Xiaolong Wang,et al.  Active Deep Networks for Semi-Supervised Sentiment Classification , 2010, COLING.

[201]  Lin Yang,et al.  Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation , 2017, MICCAI.

[202]  Qi Tian,et al.  MARS: A Video Benchmark for Large-Scale Person Re-Identification , 2016, ECCV.

[203]  Theodoros Rekatsinas,et al.  Deep Learning for Entity Matching: A Design Space Exploration , 2018, SIGMOD Conference.

[204]  David Janz,et al.  Actively Learning what makes a Discrete Sequence Valid , 2017, ArXiv.

[205]  Silvio Savarese,et al.  Active Learning for Convolutional Neural Networks: A Core-Set Approach , 2017, ICLR.

[206]  Ran El-Yaniv,et al.  Deep Active Learning with a Neural Architecture Search , 2018, NeurIPS.

[207]  Chu-Song Chen,et al.  Cross-Age Reference Coding for Age-Invariant Face Recognition and Retrieval , 2014, ECCV.

[208]  John Langford,et al.  Agnostic active learning , 2006, J. Comput. Syst. Sci..

[209]  Kazim Hanbay.  Deep neural network based approach for ECG classification using hybrid differential features and active learning , 2019, IET Signal Process..

[210]  Yonatan Belinkov,et al.  Identifying and Controlling Important Neurons in Neural Machine Translation , 2018, ICLR.

[211]  Jungo Kasai,et al.  Low-resource Deep Entity Resolution with Transfer and Active Learning , 2019, ACL.

[212]  Hwee Tou Ng,et al.  Towards Robust Linguistic Analysis using OntoNotes , 2013, CoNLL.

[213]  Patrick Doherty,et al.  Deep Learning Quadcopter Control via Risk-Aware Active Learning , 2017, AAAI.

[214]  Jeff Donahue,et al.  Large Scale Adversarial Representation Learning , 2019, NeurIPS.

[215]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[216]  Sethuraman Panchanathan,et al.  Multimodal emotion recognition using deep learning architectures , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[217]  John Langford,et al.  Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds , 2019, ICLR.

[218]  Towards Robust and Reproducible Active Learning Using Neural Networks , 2020, ArXiv.

[219]  Yuan Li,et al.  Learning how to Active Learn: A Deep Reinforcement Learning Approach , 2017, EMNLP.

[220]  Yuhong Guo,et al.  Active Instance Sampling via Matrix Partition , 2010, NIPS.

[221]  Jeffrey Xu Yu,et al.  Entity Matching: How Similar Is Similar , 2011, Proc. VLDB Endow..

[222]  Kwang In Kim,et al.  Ranking CGANs: Subjective Control over Semantic Image Attributes , 2018, BMVC.

[223]  Siegfried Wahl,et al.  Leveraging uncertainty information from deep neural networks for disease detection , 2016, Scientific Reports.

[224]  John Blitzer,et al.  Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification , 2007, ACL.

[225]  Zoubin Ghahramani,et al.  Deep Bayesian Active Learning with Image Data , 2017, ICML.

[226]  Conrad Sanderson,et al.  Biometric Person Recognition: Face, Speech and Fusion , 2008 .

[227]  Shalini Priya,et al.  Identifying Infrastructure Damage during Earthquake using Deep Active Learning , 2019, 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM).

[228]  Karen Aardal,et al.  Facility Location , 2008, Encyclopedia of Algorithms.

[229]  Lise Getoor,et al.  Link-based Active Learning , 2009, NIPS 2009.

[230]  Andreas Geiger,et al.  Are we ready for autonomous driving? The KITTI vision benchmark suite , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[231]  Eui Jin Hwang,et al.  Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs. , 2019, Radiology.

[232]  Sanjoy Dasgupta,et al.  Two faces of active learning , 2009, Theor. Comput. Sci..

[233]  Michael S. Bernstein,et al.  Deep Bayesian Active Learning for Multiple Correct Outputs , 2019, ArXiv.

[234]  Philip S. Yu,et al.  Active Learning: A Survey , 2014, Data Classification: Algorithms and Applications.

[235]  Filiberto Pla,et al.  Capsule Networks for Hyperspectral Image Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[236]  Takeo Kanade,et al.  Comprehensive database for facial expression analysis , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[237]  Burr Settles,et al.  Active Learning , 2012, Synthesis Lectures on Artificial Intelligence and Machine Learning.

[238]  Joachim Denzler,et al.  Selecting Influential Examples: Active Learning with Expected Model Output Changes , 2014, ECCV.

[239]  Qing Wang,et al.  Face Recognition via Heuristic Deep Active Learning , 2017, CCBR.

[240]  Chris Callison-Burch,et al.  Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation , 2010, ACL.

[241]  Andreas Nürnberger,et al.  The Power of Ensembles for Active Learning in Image Classification , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[242]  Shai Shalev-Shwartz,et al.  Discriminative Active Learning , 2019, ArXiv.

[243]  Gustavo Carneiro,et al.  A Bayesian Data Augmentation Approach for Learning Deep Models , 2017, NIPS.

[244]  José Bento,et al.  Generative Adversarial Active Learning , 2017, ArXiv.

[245]  Cristian Danescu-Niculescu-Mizil,et al.  Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs , 2011, CMCL@ACL.

[246]  Peng Liu,et al.  Ieee Journal of Selected Topics in Applied Earth Observations and Remote Sensing 1 Active Deep Learning for Classification of Hyperspectral Images , 2022 .

[247]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[248]  Ziping Zhao,et al.  Active Learning for Speech Emotion Recognition Using Conditional Random Fields , 2013, 2013 14th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing.

[249]  Jonathon Shlens,et al.  Conditional Image Synthesis with Auxiliary Classifier GANs , 2016, ICML.

[250]  D. Rumelhart.  Learning Internal Representations by Error Propagation, Parallel Distributed Processing , 1986 .

[251]  Vangelis Metsis,et al.  SmartFall: A Smartwatch-Based Fall Detection System Using Deep Learning , 2018, Sensors.

[252]  Chenggang Yan,et al.  Towards Better Uncertainty Sampling: Active Learning with Multiple Views for Deep Convolutional Neural Network , 2019, 2019 IEEE International Conference on Multimedia and Expo (ICME).

[253]  Lawrence O. Hall,et al.  Automatic Cell Counting using Active Deep Learning and Unbiased Stereology , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[254]  Yannis Avrithis,et al.  Rethinking deep active learning: Using unlabeled data at model training , 2019, 2020 25th International Conference on Pattern Recognition (ICPR).

[255]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[256]  Nirmalya Roy,et al.  Active Deep Learning for Activity Recognition with Context Aware Annotator Selection , 2019, KDD.

[257]  K Lehnertz,et al.  Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: dependence on recording region and brain state. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[258]  Baolin Du,et al.  Breast Cancer Histopathological Image Classification via Deep Active Learning and Confidence Boosting , 2018, ICANN.

[259]  Jianqiang Huang,et al.  Deep Active Learning for Video-based Person Re-identification , 2018, ArXiv.

[260]  Klaus C. J. Dietmayer,et al.  Deep Active Learning for Efficient Training of a LiDAR 3D Object Detector , 2019, 2019 IEEE Intelligent Vehicles Symposium (IV).

[261]  Michael I. Jordan.  Serial Order: A Parallel Distributed Processing Approach , 1997 .

[262]  Pei Zhang,et al.  Active Learning for Neural Machine Translation , 2018, 2018 International Conference on Asian Language Processing (IALP).

[263]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[264]  K. Vijay-Shanker,et al.  A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping , 2009, CoNLL.

[265]  Pascal Poupart,et al.  Deep Active Learning for Dialogue Generation , 2016, *SEMEVAL.

[266]  Gary M. Weiss,et al.  Activity recognition using cell phone accelerometers , 2011, SKDD.

[267]  Yun Fu,et al.  Deep Active Learning Through Cognitive Information Parcels , 2017, ACM Multimedia.

[268]  Li Deng,et al.  A tutorial survey of architectures, algorithms, and applications for deep learning , 2014, APSIPA Transactions on Signal and Information Processing.