Evaluating an Information Retrieval System for the Semantic Desktop using Standard Measures from Information Retrieval

Evaluation of information retrieval systems is a critical aspect of information retrieval research. New retrieval paradigms, as retrieval in the Semantic Web, present an additional challenge for system evaluation as no off-the-shelf test corpora for evaluation exist. This paper describes the approach taken to evaluate an information retrieval system built for the Semantic Desktop and demonstrates how standard measures from information retrieval research are employed for evaluation.

[1]  Emine Yilmaz,et al.  Estimating average precision with incomplete and imperfect judgments , 2006, CIKM '06.

[2]  Ellen M. Voorhees,et al.  Evaluating Evaluation Measure Stability , 2000, SIGIR 2000.

[3]  James Allan,et al.  Minimal test collections for retrieval evaluation , 2006, SIGIR.

[4]  Oh,et al.  SEAL — A Framework for Developing SEmantic PortALs , 2001 .

[5]  P. Libby.  The Scientific American , 1881, Nature.

[6]  Timothy W. Finin,et al.  Information retrieval on the semantic web , 2002, CIKM '02.

[7]  Ansgar Bernardi,et al.  Overview and Outlook on the Semantic Desktop , 2005, Semantic Desktop Workshop.

[8]  Peter Scheir,et al.  Improving Search on the Semantic Desktop using Associative Retrieval Techniques , 2007 .

[9]  Yin Yang,et al.  An enhanced model for searching in semantic portals , 2005, WWW '05.

[10]  Stefanie N. Lindstaedt,et al.  Information Retrieval on the Semantic Web - Does it exist? , 2007, LWA.

[11]  Norbert Fuhr,et al.  Information Retrieval Skriptum zur Vorlesung im SS 03 , 2003 .

[12]  Stefan Decker,et al.  The Networked Semantic Desktop , 2004, WWW Workshop on Application Design, Development and Implementation Issues in the Semantic Web.

[13]  Stefanie N. Lindstaedt,et al.  A Storyboard of the APOSDLE Vision , 2006, EC-TEL.

[14]  Stefanie N. Lindstaedt,et al.  Two Protégé plug-ins for supporting document-based ontology engineering and ontological annotation at document-level , 2007 .

[15]  Ellen M. Voorhees,et al.  Retrieval evaluation with incomplete information , 2004, SIGIR '04.