Essays on Validity of Human-Computer Interaction Experiments

Human–computer interaction (HCI) experiments have been used to understand human– computer interaction and to inform the design of interactive products. However, to be able to derive causal inferences and make generalizations from experimental outcomes, close attention must be paid to how experiments are designed. Small flaws in experimental design can pose serious threats to the validity of experimental studies. Yet only limited attention has been paid to validity in HCI experiments. Many disciplines have taken steps to understand and address validity problems, and there is a common view that HCI should simply follow these steps. In this thesis, I argue for the necessity of closer attention to experimental design from within HCI.
Drawing on empirical evidence, I illustrate the importance of replication as a source of validation and cumulative knowledge in the field. I further argue that HCI sometimes fails to fully engage with developments from other disciplines. In particular, I demonstrate that some experimental artefacts well known in psychology have been overlooked, despite their relevance to HCI. I also argue that practices imported from psychology require careful adaptation to the HCI context, as, for example, control conditions commonly used in psychological experiments do not always translate well to computerized settings. Finally, I show how poor reporting of study analyses may call into question the credibility of research findings.
Overall, this thesis aims to identify validity issues in human–computer experiments and provide recommendations for how future HCI experiments can mitigate validity threats.