Are Neural Nets Modular? Inspecting Their Functionality Through Differentiable Weight Masks

Neural networks (NNs) whose subnetworks implement reusable functions are expected to offer numerous advantages, e.g., compositionality through efficient recombination of functional building blocks, interpretability, preventing catastrophic interference by separation, etc. Understanding if and how NNs are modular could provide insights into how to improve them. Current inspection methods, however, fail to link modules to their function. We present a novel method based on learning binary weight masks to identify individual weights and subnets responsible for specific functions. This powerful tool shows that typical NNs fail to reuse submodules, becoming redundant instead. It also yields new insights into known generalization issues with the SCAN dataset. Our method also unveils class-specific weights of CNN classifiers, and shows to which extent classifications depend on them. Our findings open many new important directions for future research.