A feedforward theory of visual cortex accounts for human performance in rapid categorization

2 Primates are remarkably good at recognizing objects in cluttered natural images. The level of performance of the primate visual system and its robustness to image variability have remained unchallenged by the best computer vision systems despite decades of engineering effort. We developed a new model of the feedforward path of the ventral stream in primate visual cortex that incorporates many anatomical and physiological constraints. Its key property – in addition to supervised learning from IT cortex to higher areas – is an unsupervised learning stage that creates from natural images a large generic dictionary of tuned units from V2 to IT useful for different recognition tasks. Remarkably, these model units exhibit tuning properties consistent with the known physiology of the main visual areas. Here we report that the model can predict both the level and the pattern of performance achieved by humans on a difficult animal vs. non-animal rapid categorization task. The high performance of a feedforward, hierarchical model compared with existing computer vision systems and with human vision supports a theoretical framework for understanding properties of single neurons and cortical areas in the context of high level visual functions while suggesting a novel architecture for computer vision systems. Object recognition in cortex is mediated by the ventral visual pathway running from primary visual cortex 1 , V1, through extrastriate visual areas V2 and V4 to inferotemporal cortex 2-4 , IT (comprising PIT and AIT), and then to prefrontal cortex (PFC) which is involved in linking perception to memory and action. It is well known that recognition is possible for scenes viewed in rapid visual presentation that do not allow sufficient time for eye movements 5-10 and in the near-absence of attention 11. The hypothesis that the basic processing of information is feedforward is supported most directly by the short times required for a selective response to appear in IT cells 12. Very recent data 13 convincingly show that the activity of small neuronal populations in monkey IT, over very short time intervals (as small as 12.5 ms) and only about 100 ms after stimulus onset, contains surprisingly accurate and robust information supporting a variety of recognition tasks. While this does not rule out the use of 3 local feedback loops within an area, it does suggest that a core hierarchical feedforward architecture may be a reasonable starting point for a theory of visual cortex, aiming to explain " …