On the Feasibility of a Tag-Based Approach for Deciding Which Objects a Picture Shows: An Empirical Study

Many online platforms allow users to describe resources with freely chosen keywords, so called tags. The specific meaning of a tag as well as its specific relation to the tagged resource are left open for interpretation to the user. Although human users mostly have a fair chance at interpreting it, machines do not. An algorithmic approach for identifying descriptive tags however could prove useful for intelligent search for pictures and providing first-cut overviews over tagged picture repositories. In this paper we investigate the characteristics of the problem to decide which tags describe visible entities on a given picture. Based on a systematic user study, we are able to discuss in detail the problems involved for both humans and machines when identifying descriptive tags. Furthermore, we investigate the general feasibility of developing a tag-based algorithm tackling this question. Finally, a concrete implementation and its evaluation are presented.