DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers

Dynamic networks have shown their promising capability in reducing theoretical computation complexity by adapting their architectures to the input during inference. However, their practical runtime usually lags behind the theoretical acceleration due to inefficient sparsity. Here, we explore a hardware-efficient dynamic inference regime, named dynamic weight slicing. Instead of adaptively selecting important weight elements in a sparse way, we pre-define dense weight slices with different importance level by generalized residual learning. During inference, weights are progressively sliced beginning with the most important elements to less important ones to achieve different model capacity for inputs with diverse difficulty levels. Based on this conception, we first present dynamic slimmable network (DS-Net) by input-dependently adjusting filter numbers of convolution neural networks. By extending dynamic weight slicing to multiple dimensions in both CNNs and transformers (e.g. kernel size, embedding dimension, number of heads, etc.), we further present dynamic slice-able network (DS-Net++). Both DS-Net and DS-Net++ adaptively slice a part of network parameters for inference while keeping it stored statically and contiguously in hardware to prevent the extra burden of sparse computation. To ensure sub-network generality and routing fairness, we propose a disentangled two-stage optimization scheme. In Stage-I, a series of training techniques for dynamic supernet based on in-place bootstrapping (IB) and multi-view consistency (MvCo) are proposed to improve the supernet training efficacy. In Stage-II, sandwich gate sparsification (SGS) is proposed to assist the gate training. Extensive experiments on 4 datasets and 3 different network architectures demonstrate our DS-Net and DS-Net++ consistently outperform their static counterparts as well as state-of-the-art static and dynamic model compression methods by a large margin (up to 6.6%). Typically, DS-Net++ achieves 2-4× computation reduction and 1.62× real-world acceleration over MobileNet, ResNet-50 and Vision Transformer, with minimal accuracy drops (0.1-0.3%) on ImageNet. Code release: https://github.com/changlin31/DS-Net