Analog Coding in Emerging Memory Systems

Exponential growth in data generation and large-scale data science has created an unprecedented need for inexpensive, low-power, low-latency, high-density information storage. This need has motivated significant research into multi-level memory devices that are capable of storing multiple bits of information per device. The memory state of these devices is intrinsically analog. Furthermore, much of the data they will store, along with the subsequent operations on the majority of this data, are all intrinsically analog-valued. Ironically though, in the current storage paradigm, both the devices and data are quantized for use with digital systems and digital error-correcting codes. Here, we recast the storage problem as a communication problem. This then allows us to use ideas from analog coding and show, using phase change memory as a prototypical multi-level storage technology, that analog-valued emerging memory devices can achieve higher capacities when paired with analog codes. Further, we show that storing analog signals directly through joint coding can achieve low distortion with reduced coding complexity. Specifically, by jointly optimizing for signal statistics, device statistics, and a distortion metric, we demonstrate that single-symbol analog codings can perform comparably to digital codings with asymptotically large code lengths. These results show that end-to-end analog memory systems have the potential to not only reach higher storage capacities than discrete systems but also to significantly lower coding complexity, leading to faster and more energy efficient data storage.

[1]  K. Gopalakrishnan,et al.  Phase change memory technology , 2010, 1001.1164.

[2]  L. A. Lastras-Montao,et al.  Rewritable storage channels , 2008, 2008 International Symposium on Information Theory and Its Applications.

[3]  Tor A. Ramstad,et al.  Bandwidth compression for continuous amplitude channels based on vector approximation to a continuous subset of the source signal space , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[4]  E. Eleftheriou,et al.  Drift-Tolerant Multilevel Phase-Change Memory , 2011, 2011 3rd IEEE International Memory Workshop (IMW).

[5]  Tara N. Sainath,et al.  FUNDAMENTAL TECHNOLOGIES IN MODERN SPEECH RECOGNITION Digital Object Identifier 10.1109/MSP.2012.2205597 , 2012 .

[6]  Kenneth Rose,et al.  On Zero-Delay Source-Channel Coding , 2014, IEEE Transactions on Information Theory.

[7]  Rahul Sarpeshkar,et al.  Analog Versus Digital: Extrapolating from Electronics to Neurobiology , 1998, Neural Computation.

[8]  C. Hagleitner,et al.  Device, circuit and system-level analysis of noise in multi-bit phase-change memory , 2010, 2010 International Electron Devices Meeting.

[9]  Eero P. Simoncelli,et al.  Efficient coding of natural images with a population of noisy Linear-Nonlinear neurons , 2011, NIPS.

[10]  H.-S. Philip Wong,et al.  Joint Source-Channel Coding with Neural Networks for Analog Data Compression and Storage , 2018, 2018 Data Compression Conference.

[11]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition , 2012 .

[12]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[13]  George L. Turin,et al.  The theory of optimum noise immunity , 1959 .

[14]  G. W. Burr,et al.  Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element , 2015, 2014 IEEE International Electron Devices Meeting.

[15]  Sueli I. Rodrigues Costa,et al.  Curves on a sphere, shift-map dynamics, and error control for continuous alphabet sources , 2003, IEEE Transactions on Information Theory.

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Daniele Ielmini,et al.  Analytical model for subthreshold conduction and threshold switching in chalcogenide-based memory devices , 2007 .

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  Chung Lam,et al.  Capacity optimization of emerging memory systems: A shannon-inspired approach to device characterization , 2014, 2014 IEEE International Electron Devices Meeting.

[20]  Shimeng Yu,et al.  Metal–Oxide RRAM , 2012, Proceedings of the IEEE.

[21]  Gert Cauwenberghs,et al.  Event-driven contrastive divergence for spiking neuromorphic systems , 2013, Front. Neurosci..

[22]  B. Rimoldi Beyond the separation principle: A broader approach to source-channel coding , 2002 .

[23]  Tor A. Ramstad,et al.  Shannon-kotel-nikov mappings in joint source-channel coding , 2009, IEEE Transactions on Communications.

[24]  Nam C. Phamdo,et al.  Hybrid digital-analog (HDA) joint source-channel codes for broadcasting and robust communications , 2002, IEEE Trans. Inf. Theory.

[25]  Javier Garcia-Frías,et al.  Analog joint source channel coding over non-linear acoustic channels , 2013, 2013 47th Annual Conference on Information Sciences and Systems (CISS).

[26]  Chung Lam,et al.  Experimental demonstration of array-level learning with phase change synaptic devices , 2013, 2013 IEEE International Electron Devices Meeting.

[27]  Mohammad M. Mansour,et al.  A 640-Mb/s 2048-bit programmable LDPC decoder chip , 2006, IEEE Journal of Solid-State Circuits.

[28]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[29]  Thien Truong Nguyen Ly,et al.  Efficient Hardware Implementations of LDPC Decoders, through Exploiting Impreciseness in Message-Passing Decoding Algorithms. (Mise en oeuvre matérielle de décodeurs LDPC haut débit, en exploitant la robustesse du décodage par passage de messages aux imprécisions de calcul) , 2017 .

[30]  J. Doye,et al.  Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms , 1997, cond-mat/9803344.

[31]  Wonyong Sung,et al.  Rate-0.96 LDPC Decoding VLSI for Soft-Decision Error Correction of NAND Flash Memory , 2014, IEEE Transactions on Very Large Scale Integration (VLSI) Systems.

[32]  David Minnen,et al.  Full Resolution Image Compression with Recurrent Neural Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Radford M. Neal,et al.  Near Shannon limit performance of low density parity check codes , 1996 .

[34]  Javier Garcia-Frías,et al.  Analog Joint Source Channel Coding over Non-Linear Channels , 2013, 2013 Data Compression Conference.

[35]  M. Breitwisch Phase Change Memory , 2008, 2008 International Interconnect Technology Conference.

[36]  Y.C. Chen,et al.  Write Strategies for 2 and 4-bit Multi-Level Phase-Change Memory , 2007, 2007 IEEE International Electron Devices Meeting.

[37]  Fredrik Hekland On the Design and Analysis of Shannon-Kotel'nikov Mappings for Joint-Source-Channel Coding , 2007 .

[38]  Yichuan Hu,et al.  Analog Joint Source-Channel Coding Using Non-Linear Curves and MMSE Decoding , 2011, IEEE Transactions on Communications.

[39]  Sae-Young Chung,et al.  On the construction of some capacity-approaching coding schemes , 2000 .

[40]  Javier Garcia-Frias,et al.  Analog Joint Source Channel Coding for Wireless Optical Communications and Image Transmission , 2014, Journal of Lightwave Technology.

[41]  Luca Crippa,et al.  A 4Gb 2b/cell NAND Flash Memory with Embedded 5b BCH ECC for 36MB/s System Read Throughput , 2006, 2006 IEEE International Solid State Circuits Conference - Digest of Technical Papers.

[42]  Xin Zheng,et al.  Error-Resilient Analog Image Storage and Compression with Analog-Valued RRAM Arrays: An Adaptive Joint Source-Channel Coding Approach , 2018, 2018 IEEE International Electron Devices Meeting (IEDM).

[43]  Deniz Gündüz,et al.  Deep Joint Source-channel Coding for Wireless Image Transmission , 2018, ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[44]  Shimeng Yu,et al.  Synaptic electronics: materials, devices and applications , 2013, Nanotechnology.

[45]  M. Breitwisch,et al.  Novel Lithography-Independent Pore Phase Change Memory , 2007, 2007 IEEE Symposium on VLSI Technology.

[46]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[47]  A. Pirovano,et al.  Electrothermal and phase-change dynamics in chalcogenide-based memories , 2004, IEDM Technical Digest. IEEE International Electron Devices Meeting, 2004..

[48]  Lav R. Varshney,et al.  Optimal Information Storage in Noisy Synapses under Resource Constraints , 2006, Neuron.

[49]  Dave Evans,et al.  How the Next Evolution of the Internet Is Changing Everything , 2011 .

[50]  P.A. Floor,et al.  Optimality of Dimension Expanding Shannon-Kotel'nikov Mappings , 2007, 2007 IEEE Information Theory Workshop.

[51]  Richard H. Sherman,et al.  Chaotic communications in the presence of noise , 1993, Optics & Photonics.

[52]  Mikael Skoglund,et al.  Zero-Delay Joint Source-Channel Coding for a Bivariate Gaussian on a Gaussian MAC , 2012, IEEE Transactions on Communications.

[53]  Farnood Merrikh-Bayat,et al.  Training and operation of an integrated neuromorphic network based on metal-oxide memristors , 2014, Nature.

[54]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[55]  Mikael Skoglund,et al.  Hybrid Digital–Analog Source–Channel Coding for Bandwidth Compression/Expansion , 2006, IEEE Transactions on Information Theory.

[56]  Michael Gastpar,et al.  To code, or not to code: lossy source-channel communication revisited , 2003, IEEE Trans. Inf. Theory.

[57]  Valero Laparra,et al.  End-to-end Optimized Image Compression , 2016, ICLR.

[58]  Henrik Ekström,et al.  COMSOL Multiphysics®: Finite element software for electrochemical analysis. A mini-review , 2014 .

[59]  Sergio Verdú,et al.  Lossy Joint Source-Channel Coding in the Finite Blocklength Regime , 2012, IEEE Transactions on Information Theory.

[60]  Ilangko Balasingham,et al.  Zero Delay Joint Source Channel Coding for Multivariate Gaussian Sources over Orthogonal Gaussian Channels , 2013, Entropy.