Evolving networks: using the genetic algorithm with connectionist learning

It is appealing to consider hybrids of neural-network learning algorithms with evolutionary search procedures, simply because Nature has so successfully done so. In fact, computational models of learning and evolution ooer theoretical biology new tools for addressing questions about Nature that have dogged that eld since Darwin Belew, 1990]. The concern of this paper, however, is strictly artiicial: Can hybrids of connectionist learning algorithms and genetic algorithms produce more eecient and eeective algorithms than either technique applied in isolation? The paper begins with a survey of recent work (by us and others) that combines Holland's Genetic Algorithm (GA) with con-nectionist techniques and delineates some of the basic design problems these hybrids share. This analysis suggests the dangers of overly literal representations of the network on the genome (e.g., encoding each weight explicitly). A preliminary set of experiments that use the GA to nd unusual but successful values for BP parameters (learning rate, momentum) are also reported. The focus of the report is a series of experiments that use the GA to explore the space of initial weight values , from which two diierent gradient techniques (conjugate gradient and back propagation) are then allowed to optimize. We nd that use of the GA provides much greater conndence in the face of the stochas-tic variation that can plague gradient techniques, and can also allow training times to be reduced by as much as two orders of magnitude. Computational trade-oos between BP and the GA are considered, including discussion of a software facility that exploits the parallelism inherent in GA/BP hybrids. This evidence leads us to conclude that the GA's global sampling characteristics compliment connectionist local search techniques well, leading to eecient and reliable hybrids.