Spring 2025 GRASP SFI: Yossi Gandelsman, University of California, Berkeley, “Interpreting the Inner Workings of Vision Models”

This will be a hybrid event with in-person attendance in Levine 307 and virtual attendance on Zoom.

ABSTRACT

In this talk, I present an approach for interpreting the internal computation in deep vision models. I show that these interpretations can be used to detect model bugs and to improve the performance of pre-trained deep neural networks (e.g., reducing hallucinations from image captioners and detecting and removing spurious correlations in CLIP) without any additional training. Moreover, the obtained understanding of deep representations can unlock new model capabilities (e.g., novel identity editing techniques in diffusion models and faithful image inversion in GANs). I demonstrate how to find common representations across different models (discriminative and generative) and how deep representations can be adapted at test-time to improve model generalization without any additional supervision. Finally, I discuss future work on improving the presented interpretation techniques and their application to continual model correction and scientific discovery.