Finding the Unicorn within a Million Patches: A Material-Driven Journey into the Hidden Layers of a Diffusion Model

Today's AI models learn rich internal representations, such as the visual features inside diffusion models that produce generated images, and offer a new kind of material for co-creation. However, interfaces for creating with generative AI typically operate on the level of inputs and outputs, obscuring the material formation that unfolds in between. To address this gap, this pictorial documents a material-driven journey of designing an interface for interacting with the hidden layers of diffusion models.
Working in an interdisciplinary team of interpretability researchers and interaction designers, we explore how methods from mechanistic interpretability can inform alternative strategies that move beyond black-boxed interaction paradigms.
Through visual exploration of technical material characteristics combining experimentation and drawing, we articulate AI's internal representations as design material. Rather than hiding model properties behind conversational agents, we present a material-driven journey for building interfaces that exposes and engages with them. Finally, we discuss how this approach might enable different forms of co-creation with generative models.