Improving Anytime Prediction with Parallel Cascaded Networks and a Temporal-Difference Loss

Although deep feedforward neural networks share some characteristics with the primate visual system, a key distinction is their dynamics. Deep nets typically operate in serial stages wherein each layer completes its computation before processing begins in subsequent layers. In contrast, biological systems have cascaded dynamics: information propagates from neurons at all layers in parallel but transmission occurs gradually over time, leading to speed-accuracy trade o s even in feedforward architectures. We explore the consequences of biologically inspired parallel hardware by constructing cascaded ResNets in which each residual block has propagation delays but all blocks update in parallel in a stateful manner. Because information transmitted through skip connections avoids delays, the functional depth of the architecture increases over time, yielding anytime predictions that improve with internal-processing time. We introduce a temporal-di erence training loss that achieves a strictly superior speed-accuracy profile over standard losses and enables the cascaded architecture to outperform state-of-the-art anytime-prediction methods. The cascaded architecture has intriguing properties, including: it classifies typical instances more rapidly than atypical instances; it is more robust to both persistent and transient noise than is a conventional ResNet; and its time-varying output trace provides a signal that can be exploited to improve information processing and inference. Since the earliest investigations of artificial neural nets, their design has been informed by biological neural nets [37]. Perhaps the most compelling example is the convolutional net for machine vision, which has adopted properties of primate cortical neuroanatomy including a hierarchical layered organization, local receptive fields, and spatial equivariance [12]. In this article, we investigate computational consequences of two fundamental properties of biological information processing systems that have not been considered in the design of deep neural nets. First, the brain consists of massively parallel, dedicated hardware with neurons throughout the cortex updating continuously and simultaneously. Second, information transmission between neurons introduces time delays [1]. As a result, unrefined and possibly úCurrently at Apple. 35th Conference on Neural Information Processing Systems (NeurIPS 2021). incomplete neural state in one region is transmitted to the next region even as the state evolves; and feedforward connectivity yields a speed-accuracy trade o in which the initial response to a static input occurs rapidly but can be inaccurate, with the output gradually improving over internal processing time. Following McClelland [36], we refer to such an architecture as cascaded. Cascaded dynamics contrast sharply with the dynamics of standard feedforward nets, which operate in serial stages, each layer completing its computation before subsequent layers begin processing. Cascaded dynamics are also quite di erent than the dynamics of vision models with recurrent connections [e.g., 23, 25, 38, 47], which, given a static input, may iteratively update, but layer updates are still computed serially with each layer completing its computation and then feeding it immediately to the next layer (or back to itself). Fundamentally, our investigation asks: Supposing we take a step toward biological realism with massively parallel hardware and relatively slow inter-neuron communication, what are the computational benefits and consequences?2 We construct cascaded networks by introducing propagation delays in deep feedforward nets provided with a static input. We treat the net as massively parallel such that all units across all layers are updated simultaneously and iteratively. We focus on the ResNet architecture [14] and we introduce a propagation delay into each residual block (Figure 1a). Because the skip connection permits faster transmission of more primitive perceptual representations, the functional depth of the resulting architecture increases over internal-processing time, yielding a trade o between processing speed and complexity of processing. Consequently, the architecture o ers a natural, integral mechanism for making predictions at any point in processing, known as anytime prediction [58]. Speed-accuracy trade o s are a fundamental characteristic of human information processing [22, 42] and human perception has been modeled with deep learning anytime prediction methods [29]. Although we focus on the ResNet, our approach can be incorporated into any model with skip connections (e.g., Highway Nets [48], DenseNet [19], U-Net [43], Transformers [52]). The contrast between a serial, one-layer-at-a-time model and a cascaded, parallel-update model is illustrated in Figures 1b and 1c, respectively. To step through the operation of the cascaded model, at time 1, only the first residual block has received meaningful input, and the model prediction is therefore based only on this block’s computation. At time 2, all higher residual blocks have received input from block 1, and the output is therefore based on all blocks’ computations, though blocks 2 and above have deficient input. At each subsequent time, all blocks are receiving meaningful input, but it is not until time t that block t has reached its asymptotic output because its input does not stabilize until t ≠ 1. In essence, the cascaded model behaves like a WideResNet [56] on the first steps and then becomes a deep ResNet. Our work makes the following key contributions. • We demonstrate the superiority of the cascaded architecture to the serial (Figures 1b,c), indicating that parallelism can be exploited in a way that has not previously been studied. • We propose and evaluate a novel training objective aimed at improving the predictions of anytime models. This temporal-di erence (TD) loss [49] encourages the most accurate response as quickly as possible. TD training improves the performance of both cascaded and serial architectures. Although a rich literature exists aimed at reducing the number of computational steps required to obtain an accurate answer [2, 3, 4, 5, 11, 15, 16, 17, 18, 20, 24, 31, 31, 38, 40, 45, 51, 54, 57], all of this work uses a degenerate form of TD for training and our results suggest that these models can be improved using TD. • The cascaded model trained with TD (CascadedTD) tends to respond most rapidly to prototypical exemplars, whereas training with the standard cross-entropy loss (CascadedCE) does not (Figure 2). We assess with three quantitative prototypicality measures, and we further show that CascadedTD rapidly converges on the correct semantic family, whereas CascadedCE does not. These facts indicate that CascadedTD organizes knowledge di erently across layers than does CascadedCE. 2Like much other research in deep learning [25, 8], biology informs our work by providing novel forms of inductive bias. Our goal is to investigate computational consequences of these biases, not to model biological phenomena per se.