Third Workshop on Human-Centered Evaluation and Auditing of Language Models: AI Agents-in-the-Loop

Large Language Models (LLMs) are increasingly deployed in real-world applications but also pose significant risks. Our previous workshop iterations at CHI'24 and CHI'25 brought together HCI and AI researchers to address the “evaluation crisis” through human-centered evaluation approaches. As the demand for human-centered evaluation grows, a new frontier has emerged: practitioners are increasingly turning to AI agents themselves as tools to support evaluation processes. This third iteration introduces the theme of AI agents-in-the-loop, exploring the emerging frontier where human judgment meets agent automation in LLM evaluation workflows. The workshop will examine critical questions about task allocation between humans and agents, meta-evaluation of evaluator agents, and the design of safeguards that preserve human agency while benefiting from automation. Through position papers, keynote discussion, and collaborative activities, participants will identify key challenges, share emerging practices, and outline research directions for hybrid human-AI auditing and evaluation approaches.