Robert Bamler, Stephan Mandt
Published 2019-07-04Version 1
Continuous symmetries and their breaking play a prominent role in contemporary physics. Effective low-energy field theories around symmetry breaking states explain diverse phenomena such as superconductivity, magnetism, and the mass of nucleons. We show that such field theories can also be a useful tool in machine learning, in particular for loss functions with continuous symmetries that are spontaneously broken by random initializations. In this paper, we illuminate our earlier published work (Bamler & Mandt, 2018) on this topic more from the perspective of theoretical physics. We show that the analogies between superconductivity and symmetry breaking in temporal representation learning are rather deep, allowing us to formulate a gauge theory of `charged' embedding vectors in time series models. We show that making the loss function gauge invariant speeds up convergence in such models.
Comments: Presented at the ICML 2019 Workshop on Theoretical Physics for Deep Learning
Probing the Robustness of Independent Mechanism Analysis for Representation Learning
On Pitfalls of Identifiability in Unsupervised Learning. A Note on: "Desiderata for Representation Learning: A Causal Perspective"
The dynamics of representation learning in shallow, non-linear autoencoders
![QR code for arXiv:1907.02163 [stat.ML]](http://oss.arxitics.com/images/favicon.svg)