Learning efficient linear codes for natural images: the roles of sparseness, overcompleteness, and statistical independence

An algorithm is described which allows for the learning of sparse, overcomplete image representations. Images are modeled as a linear superposition of basis functions, and a set of basis functions is sought which maximizes the sparseness of the representation (fewest number of active units per image). When applied to natural scenes, the basis functions converge to localized, oriented, bandpass functions that bear a strong resemblance to the receptive fields of neurons in the primate striate cortex. Importantly, the code can be made overcomplete, which allows for an increased degree of sparseness in which the basis functions can become more specialized. The learned basis functions constitute an efficient representation of natural images because sparseness forces a form of reduced entropy representation that minimizes statistical dependencies among outputs.