Abstract: I will outline challenges and shifting paradigms in continual learning (CL) within the context of large models. I emphasize stability-plasticity dilemma that CL must balance catastrophic forgetting with adapting to new knowledge. By four key issues, I will show how conventional setups fail to ensure fair computational comparisons and explore the complexities of managing CL across heterogeneous network architectures. Furthermore, I present evidence that for large foundation models, knowledge acquisition and adaptation become more dominant challenges than forgetting itself. Ultimately, I argue that advancing large-scale CL requires a shift toward innovative approaches, such as decentralized collaboration frameworks and modular memory systems

Bio: I received my B.S. (2003) and M.S. (2008) in Electrical and Computer Engineering from Seoul National University, and my Ph.D. (2015) from the University of Maryland, College Park, under the supervision of Prof. Larry S. Davis. Since 2024, I have been an Associate Professor in the Department of Electrical and Computer Engineering at SNU. During my Ph.D., I did research internships at the US Army Research Lab (2012), Adobe Research (2013), Disney Research Pittsburgh (2014), and Microsoft Research Redmond (2014). I subsequently worked as a Research Scientist at the Allen Institute for AI (AI2, Seattle, 2016–2018) and at Comcast Applied AI Research (Washington, D.C., 2015–2016), and held faculty positions as Assistant Professor at GIST (2018–2022) and Associate Professor at Yonsei University (2022–2024). I serve as an Area Chair for NeurIPS, ICML, ICLR, CVPR, WACV, AAAI (Senior PC), and ICRA, and as an Associate Editor for IEEE Transactions on PAMI and IJCV. My research focuses on visual understanding for edge devices and home robots, and on image/video visual recognition under computational and data-efficiency constraints.

Abstract: Foundation models like GPT-4 have dramatically altered the modern work landscape for many industries reliant on language tasks, but no equivalent model exists yet for scientific applications. Incorporating foundation models into research workflows could enable unprecedented discoveries that connect traditionally distinct scientific sub-disciplines. However, mainstream foundation models trained on human-scale datasets will be insufficient for analyzing most scientific phenomena -- a foundation model for science will require special consideration for the requirements of scientific datasets, especially those with wide dynamic ranges. In this talk, I will introduce the Polymathic AI initiative: our goal is to accelerate the development of versatile foundation models tailored for numerical datasets and scientific machine learning tasks. The challenge we are undertaking is to build AI models which leverage information from heterogeneous datasets and across different scientific fields, which, contrary to domains like natural language processing, do not share a unifying representation (i.e., text). Such models can then be used as strong baselines or be further fine-tuned by scientists for specific applications. This approach has the potential to democratize AI in science by providing off-the-shelf models that have stronger priors (i.e., background knowledge) for shared general concepts such as causality, measurement, signal processing, and even more specialized shared concepts like wave-like behavior, which otherwise would need to be learned from scratch. I will present our initial papers and projects, including large scientific datasets designed for large scale training "MultiModal Universe" and "The Well".

Bio: Dr. Shirley Ho is a Professor of Physics and Data Science at New York University, Group Leader of Cosmology and Machine Learning at Simons Foundation with visiting appointments at Princeton University and Carnegie Mellon University. She is also co-Founder and Director of Polymathic AI, a research collective aiming at advancing science through multi-disciplinary AI. Dr. Shirley Ho is noted for her work in leading the early adoption of Artificial Intelligence in Astrophysics. In particular, her team at Carnegie Mellon University was the first to apply 3D convolutional neural network in astrophysics, the same team then accelerated astrophysical simulations with deep learning for the first time. Her current team at Simons Foundation, NYU and Princeton is the first to combine symbolic regression and neural network to recover physical laws from observations directly. Her team also led the first development and deployment of deep learning accelerated simulation based inference framework for large astronomical surveys. Her team further accelerated physical simulations ranging from fluid dynamics simulations to planetary dynamics simulations using modern deep learning techniques, and developed techniques in interpretable machine learning for science. Dr. Ho has won several prizes for her significant contributions to the fields of astrophysics and AI. The list includes Schmidt Sciences AI2050 Senior Fellow, National Blavatnik Award Finalist, European Physical Society Giuseppe and Vanna Cocconi Prize in cosmology, NASA Group Achievement Award, Macronix Prize, Carnegie Science Award. She was also elected as a Fellow of the International Astrostatistics Association in 2020. She served on advisory boards at NASA, DOE, National Academies and numerous NSF institutes. She always tries to balance between Machine learning, the Universe, and life, and almost always fails to do so.

Abstract: Bayesian inference admits a variational form: the posterior minimizes expected loss regularized by the KL divergence to the prior. Recent generalizations vary the loss, divergence, and variational family independently, which can yield inconsistent objectives (e.g., a Gaussian family against a bounded-support prior results in infinite KL divergence). We instead couple these ingredients through a single Fenchel-Young term built from a convex regularizer, ruling out inconsistent combinations. Choosing a Tsallis negentropy as the regularizer, posteriors become deformed exponential families with adaptive support or heavy tails, recovering standard Bayes at the Shannon limit. Conjugacy and additive canonical updates are preserved, yielding generalized versions of EM, VAE, and Bayesian-learning-rule algorithms.

Bio: Mário Figueiredo is an IST Distinguished Professor and holder of the Feedzai Chair on Machine Learning at Instituto Superior Técnico (IST), University of Lisbon. He also serves as a group leader at Instituto de Telecomunicações and is the Director of the ELLIS Unit Lisbon. His research career spans a broad range of topics, including statistical machine learning, signal processing, optimization, and inverse problems. He is a Fellow of IEEE, IAPR, EURASIP, and ELLIS, and member of the Lisbon Academy of Sciences and the Portuguese Academy of Engineering.

Abstract: Importance weighted variational inference (VI) approximates densities known up to a normalizing constant by optimizing bounds that tighten with the number of Monte Carlo samples N. Standard optimization relies on reparameterized gradient estimators, which are well-studied theoretically yet restrict both the choice of the data-generating process and the variational approximation. While REINFORCE gradient estimators do not suffer from such restrictions, they lack rigorous theoretical justification. In this paper, we provide the first comprehensive analysis of REINFORCE gradient estimators in importance weighted VI, leveraging this theoretical foundation to diagnose and resolve fundamental deficiencies in current state-of-the-art estimators. Specifically, we introduce and examine a generalized family of variational inference for Monte Carlo objectives (VIMCO) gradient estimators. We prove that state-of-the-art VIMCO gradient estimators exhibit a vanishing signal-to-noise ratio (SNR) as N increases, which prevents effective optimization. To overcome this issue, we propose the novel VIMCO-⋆ gradient estimator and show that it averts the SNR collapse of existing VIMCO gradient estimators by achieving a √N SNR scaling instead. We demonstrate its superior empirical performance compared to current VIMCO implementations in challenging settings where reparameterized gradients are typically unavailable.

Bio: Kamélia Daudel is an Assistant Professor of Statistics at ESSEC Business School. Prior to that, she was a postdoctoral researcher in the Department of Statistics at the University of Oxford working with Arnaud Doucet. Her research lies in the field of Bayesian Inference. In particular, she is interested in Variational Inference methods which go beyond the commonly-used parametric variational distribution framework and which involve flexible variational bounds.