Injecting Realism into Simulated Work Tasks: A Case Study of the Book Domain

Interactive IR studies are commonly conducted by assigning participants simulated work tasks situations (SWTS), predetermined textual descriptions of an information need and its context to motivate participants to perform realistic searches of the IR system in question. Despite their popularity, research has shown that formulating realistic and engaging SWTSs remains challenging. In this paper, we propose using the analyses of large samples of complex real-world search requests—with the specific case of book search—to inject realism into SWTSs by taking different relevance aspects and their relative frequencies into account when formulating them. This should make it easier for researchers to design realistic SWTSs in the future.