Contextual information retrieval has promised much and has produced interesting ideas and new systems developments. Although there has been much discussion on the evaluation of context-free information retrieval systems, there has been far less discussion on the challenges of evaluating contextual systems.

In this presentation I will step back from the debates about speci c methodologies to argue for approaches that acknowledge the context of use of IR systems and the context of information itself. I shall pose some challenges for contextual IR evaluation based on the context in which our systems are deployed in real-life situations and give a personal view on some useful evaluation directions.