Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation
Ilan Naiman*, Nimrod Berman*, and Omri Azencot. * joint first authors ICML, 2023. Abstract Unsupervised disentanglement is a long-standing challenge in representation learning. Recently, self-supervised techniques achieved impressive results in the sequential setting, where data is time-dependent. However, the latter methods employ modality-based data augmentations and random sampling or solve auxiliary tasks. In this work, we propose to avoid that by generating, sampling, and comparing empirical distributions from the underlying variational model. Unlike existing work, we introduce a self-supervised sequential disentanglement framework based on contrastive estimation with \emph{no} external signals, while using common batch sizes and samples from the latent…
Data Representations’ Study of Latent Image Manifolds
Ilya Kaufman, and Omri Azencot. ICML, 2023. Abstract Deep neural networks have been demonstrated to achieve phenomenal success in many domains, and yet their inner mechanisms are not well understood. In this paper, we investigate the curvature of image manifolds, i.e., the manifold deviation from being flat in its principal directions. We find that state-of-the-art trained convolutional neural networks for image classification have a characteristic curvature profile along layers: an initial steep increase, followed by a long phase of a plateau, and followed by another increase. In contrast, this behavior does not appear in untrained networks in which the curvature…
Multifactor Sequential Disentanglement via Structured Koopman Autoencoders
Nimrod Berman*, Ilan Naiman*, and Omri Azencot. * joint first authors ICLR, 2023. notable-top-25%! Abstract Disentangling complex data to its latent factors of variation is a fundamental task in representation learning. Existing work on sequential disentanglement mostly provides two factor representations, i.e., it separates the data to time-varying and time-invariant factors. In contrast, we consider multifactor disentanglement in which multiple (more than two) semantic disentangled components are generated. Key to our approach is a strong inductive bias where we assume that the underlying dynamics can be represented linearly in the latent space. Under this assumption, it becomes natural to exploit…
An Operator Theoretic Approach for Analyzing Sequence Neural Networks
Ilan Naiman*, and Omri Azencot AAAI, 2023. Abstract Analyzing the inner mechanisms of deep neural networks is a fundamental task in machine learning. Existing work provides limited analysis or it depends on local theories, such as fixed-point analysis. In contrast, we propose to analyze trained neural networks using an operator theoretic approach which is rooted in Koopman theory, the Koopman Analysis of Neural Networks (KANN). Key to our method is the Koopman operator, which is a linear object that globally represents the dominant behavior of the network dynamics. The linearity of the Koopman operator facilitates analysis via its eigenvectors and…
A Differential Geometry Perspective on Orthogonal Recurrent Models
Omri Azencot, Benjamin Erichson, Mirela Ben-Chen, and Michael Mahoney. arXiv, 2021. Abstract Recently, orthogonal recurrent neural networks (RNNs) have emerged as state-of-the-art models for learning long-term dependencies. This class of models mitigates the exploding and vanishing gradients problem by design. In this work, we employ tools and insights from differential geometry to offer a novel perspective on orthogonal RNNs. We show that orthogonal RNNs may be viewed as optimizing in the space of divergence-free vector fields. Specifically, based on a well-known result in differential geometry that relates vector fields and linear operators, we prove that every divergence-free vector field is…