Avik Roy, Postdoc working on the FAIR4HEP project within CAII presented at the IRIS-HEP Meeting on March 14, 2022 with a focus area on Interpretability of AI models and FAIR for AI. Roy presented on “Exploring Interpretability of Neural Networks in the Context of FAIR principles for AI Models.”
Abstract: While neural networks are traditionally regarded as black box nonlinear functional surrogates, their interpretability is crucial to ensure model reliability and reusability. This talk will explore the relationship between FAIR (Findable, Accessible, Interoperable, Reusable) principles and interpretability of AI models. It will look into feature importance and neuron activity in the context of an Interaction Network model trained to distinguish boosted H->bb jets from QCD backgrounds. We will explore a number of approaches to probe the inside of the underlying Neural Network to understand its activity for different jet classes and how that information can be used to optimize model architecture without compromising its performance.