Analogies Emerge from Learning Dyamics in Neural Networks

When a neural network is trained on multiple analogous tasks, previous research has shown that it will often generate representations that reflect the analogy. This may explain the value of multi-task training, and also may underlie the power of human analogical reasoning – awareness of analogies may emerge naturally from gradient-based learning in neural networks. We explore this issue by generalizing linear analysis techniques to explore two sets of analogous tasks, show that analogical structure is commonly extracted, and address some potential implications.

[1]  L. Mirsky.  SYMMETRIC GAUGE FUNCTIONS AND UNITARILY INVARIANT NORMS , 1960 .

[2]  K. Holyoak,et al.  Analogical problem solving , 1980, Cognitive Psychology.

[3]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[4]  D. Detterman.  The case for the prosecution: Transfer as an epiphenomenon. , 1996 .

[5]  Daniel L. Schwartz,et al.  Rethinking transfer: A simple proposal with multiple implica-tions , 1999 .

[6]  Gerry Altmann,et al.  Mapping across Domains Without Feedback: A Neural Network Model of Transfer of Implicit Knowledge , 1999, Cogn. Sci..

[7]  Dedre Gentner,et al.  Why we’re so smart , 2003 .

[8]  James L. McClelland,et al.  A simple model from a powerful framework that spans levels of analysis , 2008, Behavioral and Brain Sciences.

[9]  Robert L. Goldstone,et al.  Analogical transfer from a simulated physical system. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[10]  James L. McClelland,et al.  Context, cortex, and associations: a connectionist developmental approach to verbal analogies , 2013, Front. Psychol..

[11]  Surya Ganguli,et al.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , 2013, ICLR.

[12]  Dianhai Yu,et al.  Multi-Task Learning for Multiple Language Translation , 2015, ACL.

[13]  Razvan Pascanu,et al.  Policy Distillation , 2015, ICLR.

[14]  Quoc V. Le,et al.  Multi-task Sequence to Sequence Learning , 2015, ICLR.

[15]  Martin Wattenberg,et al.  Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation , 2016, TACL.