3 3 A Computational Model of Avian Song Learning

Oscine song learning has an auditory phase during which a tutor song is learned and a sensorimotor phase of successive improvement that leads to adult song. A theoretical framework for song learning is presented based on the hypothesis that the primary role of the anteriorforebrainpathway of the song system is to transform an auditory template to a motor program by a form of reinforcement learning. This framework was tested by building a network model of the song-learning system including a model of the syrinx, the avian vocal organ. The model replicated the spectral envelopes of the syllables from zebra finch songs after several hundred trials of learning. The performance of the model was even better when trained on songs generated by another model having the same architecture. Experiments are proposed to further test the biological plausibility of the hypothesis, which may lead to a more detailed model of the song-learning system. Other types of sensorimotor learning based on mimicry could be implemented with a similar type of computational model. In comparison with our understanding of the preprogrammed central pattern generators found in many invertebrates and lower vertebrates responsible for complex motor behaviors (Cohen, Rossignol, and Grillner, 1988; Harris-Wanick et al., 1992; Kristan, 1992), much less is known about the representation of motor patterns acquired through experience in humans and other vertebrates, such as walking, riding a bicycle, or talking. Singing in oscine birds is a favorable system for studying the acquisition of complex motor patterns. Much is known about the ethology of birdsong learning and the influence of early auditory learning (Marler, 1963; Konishi, 1965; Marler, 1991; Catchpole and Slater, 1995). The major brain nuclei involved in song control and learning have been identified, as schematically shown in figure 33.1 (Nottebohm, Stokes, and Leonard, 1976; Bottjer et al., 1989). New data are accumulating from lesion and recording experiments on these nuclei (for reviews, see Konishi, 1985; Doupe, 1993; Margoliash, 1997; Bottjer and Arnold, 1997). KENJI DOYA Kawato Dynamic Brain Project, ERATO, Japan Science and Technology Corporation, Kyoto, Japan, and Howard Hughes Medical Institute, Salk Institute for Biological Studies, LaJolla, Calif. TERRENCE J. SEJNOWSKI Department of Biology, University of California, San Diego, and Howard Hughes Medical Institute, Salk Institute for Biological Studies, La Jolla, Calif. The primary goal of this chapter is to present a theoretical framework and a working model for song learning based on recent experimental findings. Specifically, we focus on the function of the anteriorforebrain pathway, which is not involved in song production in adult birds, but is necessary for song learning in young birds (Bottjer, Miesner, and Arnold, 1984). Our main hypothesis is that the anterior forebrain pathway works as a reinforcement learning system that is similar to the adaptive critic architecture proposed by Barto, Sutton, and Anderson (1983). The song template is a key concept in birdsong learning. A young male bird listens to a tutor song during the critical period and memorizes a template of the song; later, the bird learns to sing the stored song by comparing its own vocalization to the song template using auditory feedback (Konishi, 1965). However, it is still an open question how the song is encoded and where in the bird's brain the song template is stored. Recent experiments in zebra finch suggest that the song control system has a hierarchical organization: HVc, the high vocal center, is involved in producing a sequence of syllables, whereas its downstream nucleus RA is responsible for the subsyllabic components (Vu, Mazurek, and Kuo, 1994; Yu and Margoliash, 1996). If we assume that a song is learned in such a hierarchical fashion, the problem of song learning can be decomposed into the following three subproblems: 1. Sensory encoding: How to encode the acoustic features of syllables in such a way that they are reliably recognized. 2. Sequential memory: How to organize the network so that syllable sequences are stably memorized and reproduced. 3. Motor decoding: How to find the motor command patterns needed to replicate the acoustic features of each syllable. Existing experimental evidence does not provide straightforward solutions to these problems. A computational approach could help in exploring the biological solutions and, in particular, in providing functional + Anterior Forebrain Pathway respiratory system FIGURE 33.1 Schematic diagram of the major songbird brain nuclei involved in song control. The thinner arrows show the direct motor control pathway, and the thicker arrows show the anterior forebrain pathway. Abbreviations: Uva, nucleus uvaeformis of thalamus; NIf, nucleus interface of neostriatum; L, field L of forebrain; HVc, high vocal center (formerly called hyperstriatum ventrale, pars caudale); RA, robust nucleus of archistriatum; DM, dorsomedial part of nucleus intercollicularis; nXIIts, tracheosyringeal part of hypoglossal nucleus; AVT, ventral area of Tsai of midbrain; X, area X of lobus parolfactorius; DLM, medial part of dorsolateral nucleus of thalamus; LMAN, lateral magnocellular nucleus of anterior neostriatum. constraints on the organization of the learning system. For example, theories of unsupervised learning (von der Malsburg, 1973; Amari, 1977; Linsker, 1986; Bell and Sejnowski, 1995) suggest several possible solutions to sensory encoding problems. Studies of associative memory networks (Fukushima, 1973; Sompolinsky and Kanter, 1986; Dehaene, Changeux, and Nadal, 1987; Amari, 1988; Morita, 1996) provide constraints on representation and architectures that enable stable storage of temporal sequences. There have been extensive studies on the "inverse problem" of finding the control input for a nonlinear system to realize a given target output (Miller, Sutton, and Werbos, 1990; Gullapalli, 1995). In this chapter, we propose a working hypothesis for the functions subserved by song-related brain nuclei in songbirds (figure 33.1), with an emphasis on the role of anterior forebrain pathway in solving the motor decoding problem. Figure 33.2 illustrates various schemes for solving inverse problems using neural networks. In the first scheme (figure 33.2a), the desired output is converted to a desired motor command by an inverse model of the motor system that enables replication of the desired output in one shot. Although attractive as a model of vocal learning in other species like humans, this is not an appropriate model for vocal learning in songbirds because they require many repetitions of singing trials with auditory feedback. Another possible scheme is error correction learning (figure 33.213) that uses a linear approximation of the inverse model to convert motor output error into the motor command error for incremental learning of the control network. The problem is that the learning schemes proposed to date either use a biologically implausible algorithm (Jordan and Rumelhart, 1992) or assume the preexistence of an approximate inverse model (Kawato, Furukawa, and Suzuki, 1987; Kawato, 1990). Furthermore, in order to calculate the error in the acoustic output, a replica of the target output, or the tutor song, has to be available. The third scheme (figure 33.2~) is based on the paradigm of reinforcement learning (Sutton and Barto, 1998). It does not use an inverse model and uses a critic that evaluates the motor output by comparing the present vocal output with the tutor song. Learning is based on the correlation between stochastic changes in the motor command and the increase or decrease in the evaluation (Barto, Sutton, and Anderson, 1983; Gullapalli, 1995). There is no need to maintain a replica of the tutor song. Activation levels of auditory neurons that have selective tuning to the tutor song can be used as the