Cortical microcircuits as gated-recurrent neural networks

Cortical circuits exhibit intricate recurrent architectures that are remarkably similar across different brain areas. Such stereotyped structure suggests the existence of common computational principles. However, such principles have remained largely elusive. Inspired by gated-memory networks, namely long short-term memory networks (LSTMs), we introduce a recurrent neural network in which information is gated through inhibitory cells that are subtractive (subLSTM). We propose a natural mapping of subLSTMs onto known canonical excitatory-inhibitory cortical microcircuits. Our empirical evaluation across sequential image classification and language modelling tasks shows that subLSTM units can achieve similar performance to LSTM units. These results suggest that cortical circuits can be optimised to solve complex contextual problems and proposes a novel view on their computational function. Overall our work provides a step towards unifying recurrent networks as used in machine learning with their biological counterparts.

[1]  R. Froemke.  Plasticity of cortical excitatory-inhibitory balance. , 2015, Annual review of neuroscience.

[2]  Yun Wang,et al.  Synaptic connections and small circuits involving excitatory and inhibitory neurons in layers 2-5 of adult rat and cat neocortex: triple intracellular recordings and biocytin labelling in vitro. , 2002, Cerebral cortex.

[3]  H. Sompolinsky,et al.  Chaos in Neuronal Networks with Balanced Excitatory and Inhibitory Activity , 1996, Science.

[4]  W. Gerstner,et al.  Optimal Control of Transient Dynamics in Balanced Networks Supports Generation of Complex Movements , 2014, Neuron.

[5]  W. Gerstner,et al.  Triplets of Spikes in a Model of Spike Timing-Dependent Plasticity , 2006, The Journal of Neuroscience.

[6]  P. Strata,et al.  Dale’s principle , 1999, Brain Research Bulletin.

[7]  Rui Ponte Costa,et al.  Unified pre- and postsynaptic long-term plasticity enables reliable and flexible learning , 2015, eLife.

[8]  Andreas Lüthi,et al.  Disinhibition, a Circuit Mechanism for Associative Learning and Memory , 2015, Neuron.

[9]  K. Harris,et al.  Cortical connectivity and sensory coding , 2013, Nature.

[10]  Hilbert J. Kappen,et al.  Short-term synaptic plasticity and heterogeneity in neural systems , 2013 .

[11]  H. Markram,et al.  Interneurons of the neocortical inhibitory system , 2004, Nature Reviews Neuroscience.

[12]  M. Scanziani,et al.  Equalizing Excitation-Inhibition Ratios across Visual Cortical Neurons , 2014, Nature.

[13]  L. Abbott,et al.  Synaptic plasticity: taming the beast , 2000, Nature Neuroscience.

[14]  Wulfram Gerstner,et al.  Neuronal Dynamics: From Single Neurons To Networks And Models Of Cognition , 2014 .

[15]  Konrad P. Körding,et al.  Toward an Integration of Deep Learning and Neuroscience , 2016, bioRxiv.

[16]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[17]  Wulfram Gerstner,et al.  Spiking Neuron Models , 2002 .

[18]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[19]  Michael J. Frank,et al.  Making Working Memory Work: A Computational Model of Learning in the Prefrontal Cortex and Basal Ganglia , 2006, Neural Computation.

[20]  L.F. Abbott,et al.  Gating Multiple Signals through Detailed Balance of Excitation and Inhibition in Spiking Networks , 2009, Nature Neuroscience.

[21]  C. Koch,et al.  Persistent Single-Neuron Activity during Working Memory in the Human Medial Temporal Lobe , 2017, Current Biology.

[22]  T. Vogels,et al.  Synaptic Transmission Optimization Predicts Expression Loci of Long-Term Plasticity , 2017, Neuron.

[23]  Alexander S. Ecker,et al.  Principles of connectivity among morphologically defined cell types in adult neocortex , 2015, Science.

[24]  Christof Koch,et al.  Shunting Inhibition Does Not Have a Divisive Effect on Firing Rates , 1997, Neural Computation.

[25]  D. Sculley,et al.  Google Vizier: A Service for Black-Box Optimization , 2017, KDD.

[26]  Razvan Pascanu,et al.  On the difficulty of training recurrent neural networks , 2012, ICML.

[27]  Yoshua Bengio,et al.  Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies , 2001 .

[28]  Karl J. Friston,et al.  Canonical Microcircuits for Predictive Coding , 2012, Neuron.

[29]  Sander W. Keemink,et al.  Behavioral-state modulation of inhibition is context-dependent and cell type specific in mouse visual cortex , 2016, eLife.

[30]  Christoph E. Schreiner,et al.  Inhibitory Actions Unified by Network Integration , 2015, Neuron.

[31]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[32]  P. Heil,et al.  Persistent neural activity in auditory cortex is related to auditory working memory in humans and nonhuman primates , 2016, eLife.

[33]  Wulfram Gerstner,et al.  Spiking Neuron Models: Single Neurons, Populations, Plasticity , 2002 .

[34]  D. Hassabis,et al.  Neuroscience-Inspired Artificial Intelligence , 2017, Neuron.

[35]  Nicolas Brunel,et al.  Neuronal Dynamics , 2009, Encyclopedia of Complexity and Systems Science.

[36]  Matthew W Self,et al.  Layer-specificity in the effects of attention and working memory on activity in primary visual cortex , 2017, Nature Communications.

[37]  Mark C. W. van Rossum,et al.  Recurrent networks with short term synaptic depression , 2009, Journal of Computational Neuroscience.

[38]  M. Alexander,et al.  Principles of Neural Science , 1981 .

[39]  Shimon Whiteson,et al.  LipNet: Sentence-level Lipreading , 2016, ArXiv.

[40]  K. Harris,et al.  Laminar Structure of Spontaneous and Sensory-Evoked Population Activity in Auditory Cortex , 2009, Neuron.

[41]  Richard Socher,et al.  Pointer Sentinel Mixture Models , 2016, ICLR.

[42]  Kevan A. C. Martin,et al.  A Canonical Microcircuit for Neocortex , 1989, Neural Computation.

[43]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[44]  Konrad P. Kording,et al.  Towards an integration of deep learning and neuroscience , 2016, bioRxiv.

[45]  Thomas K. Berger,et al.  Heterogeneity in the pyramidal network of the medial prefrontal cortex , 2006, Nature Neuroscience.

[46]  Geoffrey E. Hinton,et al.  A Simple Way to Initialize Recurrent Networks of Rectified Linear Units , 2015, ArXiv.

[47]  Wulfram Gerstner,et al.  Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. , 2005, Journal of neurophysiology.

[48]  Christian K. Machens,et al.  Efficient codes and balanced networks , 2016, Nature Neuroscience.

[49]  Brent Doiron,et al.  Subtractive and Divisive Inhibition: Effect of Voltage-Dependent Inhibitory Conductances and Noise , 2001, Neural Computation.

[50]  Mark C. W. van Rossum,et al.  Probabilistic inference of short-term synaptic plasticity in neocortical microcircuits , 2013, Front. Comput. Neurosci..

[51]  Georg B. Keller,et al.  Learning Enhances Sensory and Multiple Non-sensory Representations in Primary Visual Cortex , 2015, Neuron.

[52]  Everton J. Agnes,et al.  Inhibitory Plasticity: Balance, Control, and Codependence. , 2017, Annual review of neuroscience.

[53]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[54]  Everton J. Agnes,et al.  Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks , 2015, Nature Communications.

[55]  Grace W. Lindsay,et al.  Parallel processing by cortical inhibition enables context-dependent behavior , 2016, Nature Neuroscience.

[56]  S. Prescott,et al.  Gain control of firing rate by shunting inhibition: Roles of synaptic noise and dendritic saturation , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[57]  Sen Song,et al.  Highly Nonrandom Features of Synaptic Connectivity in Local Cortical Circuits , 2005, PLoS biology.

[58]  P. Goldman-Rakic.  Cellular basis of working memory , 1995, Neuron.

[59]  Beatriz E. P. Mizusaki,et al.  Functional consequences of pre- and postsynaptic expression of synaptic plasticity , 2016, bioRxiv.

[60]  Alex Graves,et al.  Generating Sequences With Recurrent Neural Networks , 2013, ArXiv.

[61]  M. Hasselmo,et al.  Graded persistent activity in entorhinal cortex neurons , 2002, Nature.

[62]  P. Dayan,et al.  Flexible shaping: How learning in small steps helps , 2009, Cognition.

[63]  Ad Aertsen,et al.  Gating of Signal Propagation in Spiking Neural Networks by Balanced and Correlated Excitation and Inhibition , 2010, The Journal of Neuroscience.

[64]  Nicolas Brunel,et al.  Dynamics of Sparsely Connected Networks of Excitatory and Inhibitory Spiking Neurons , 2000, Journal of Computational Neuroscience.

[65]  C. Koch,et al.  Recurrent excitation in neocortical circuits , 1995, Science.

[66]  K. Harris,et al.  Population coding of tone stimuli in auditory cortex: dynamic rate vector analysis , 2009, The European journal of neuroscience.

[67]  C. Schreiner,et al.  A synaptic memory trace for cortical receptive field plasticity , 2007, Nature.

[68]  Yoshua Bengio,et al.  Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , 2014, ArXiv.

[69]  Thomas L. Dean,et al.  Neural Networks and Neuroscience-Inspired Computer Vision , 2014, Current Biology.

[70]  Shimon Whiteson,et al.  LipNet: End-to-End Sentence-level Lipreading , 2016, 1611.01599.

[71]  Mriganka Sur,et al.  Response-dependent dynamics of cell-specific inhibition in cortical networks in vivo , 2014, Nature Communications.

[72]  Alex Graves,et al.  Conditional Image Generation with PixelCNN Decoders , 2016, NIPS.

[73]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[74]  U. Bhalla.  Dendrites, deep learning, and sequences in the hippocampus , 2019, Hippocampus.

[75]  B. McNaughton,et al.  Packet-based communication in the cortex , 2015, Nature Reviews Neuroscience.

[76]  Henry Markram,et al.  An Algorithm for Modifying Neurotransmitter Release Probability Based on Pre- and Postsynaptic Spike Timing , 2001, Neural Computation.